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The National Center for Education Statistics has begun a thorough review of it 
elementary and secondary data collection program. This review is to address 
questions of the suitability, scope, reliability, and timeliness of our 
statistical data in terms of the purposes these data are expected to 
sorve—-purposes relative to instructional and administrative needs, and to 
education policy issues. The product of this effort can be the design of a ne 
plan for national data collections from institutions and individuals to be 
implemented over the next several years. 

This document is a draft of the first of two volumes that will be produced by 
the REDESIGN Project and is based on the papers and comments from the educatic 
community. To initiate the review, NCES invited a number of individuals and 
organizations to submit papers on general issues of elementary and secondary 
education data needs. This draft of the first volume synthesizes the papers v 
received by July 31, 1985. We intend to incorporate reactions to this draft i 
a later version. 

Volume 2 will present alternatives for the NCES data collection agenda 
responding to the Synthesis of Invited Papers. 

Also, the REDESIGN Project will provide an opportunity for direct public comme 
on the suggested alternatives for data acquisition and/or modifications to the 
current NCES projects. The various sectors of the education community, and 
public at large, will be invited to participate in a series of public 
discussions to be held in various cities in the Nation this school year. 

We expect the advice we are receiving will lead to substantial changes in our 
data collection efforts with regard to both their content and manner of 

collection. Of course, the Department of Education will not necessarily adopt 

every facet of the proposed alternatives for the NCES data collection agenda, 
recommendations received during the period of public comment. Indeed, it cou: 
not, since some of the advice is contradictory and some alternatives exclude 
others. 

Nor are we awaiting the ''end" of the REDESIGN Project to make needed changes. 
Our invitation for advice on our data programs lias already strengthened NCES 1 £ 
statistical program. In April, Professor Marshall Smith of the University of 
VJisconsin-Madison sent a "first installment" of his redesign paper, writing: 

"As you finalize plans for the next High School Survey, X hope that 
you do not reject, as out of hand due to cost, the idea of collecting da J 

on 8th graders and then following them through high school ..» I am more 

and more impressed with the difficulty of understanding what is going on 
in high school with data that starts in the 10th grade. Work on dropouts 
is severely limited by the 10th grade constraint ... a substantial numbe: 
of the important questions that analysts hope to address with this kind < 
survey data have to be altered and fit into studies which cannot contain 
critical information because it has not been collected." 


received from statisticians, researchers and other data users. me mimi 
our decision to shift the population for the 190U cohort from LOt:h It; 
grade students. 

We are also changing our relation with the primary suppliers ol admin is Li. 
data. Many users of the data we request from Stale education again; Inn \.o 
that the data lack comparability. In response, as one ol the l: Li st. compou 
the redesign project, we asked the Council of Chief State School nji.l.cnr.u 
undertake an assessment of the barriers to provision of comparable da l a b; 
State. A three-year contract with the Council is now In place and work Li 
progressing toward the goal of providing a broader bauu ol. comparable dab 
year. 

Now we invite you, also, to participate and contribute. butaLlu about Inn 
participate are in the opening chapter of this volume. Wo Look lorwuid la 
hearing from you. 


Kmoruou d. Klliott 
Administrater 
September 1 'JH‘> 
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PREFACE 


According to Webster, a synthesis is "the combining of diverse conception 
into a coherent whole; also, the complex so formed." 

The purpose of the Synthesis report is to represent the breadth and 
content of the invited papers in a concise, coherent manner. In this way 
participants are not required to read the set of invited papers, totallin 
over 1,000 pages at this time. 

The team of six responsible for this document met weekly, before any 
papers were received, to anticipate a structure for producing a coherent 
representation of what the papers would say. Some meetings involved 
persons who, in time, would be authors, so as to get a sense of what migh 
be in the papers. Several draft outlines later — after half the papers 
were received — the working version that produced this volume 
emerged, and chapter assignments made. All six team members read all of 
the papers more than once within the period of two months beginning with 
the arrival of the first papers in mid-June and ending with the completio 
of the final drafts in mid-August. 

The team operated under a constraint to reflect, not add to, what the 
various authors were saying. Part III of this report was added to permit 
the contractor synthesizers to make their own statements for the project 
record. Finally, the team agreed to use quotations to a great extent in 
order to communicate best the intentions of the invited paper writers. 
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Introduction 



REDESIGN PROJECT is — a dialogue to identify the data needs of the 

public and the education community at all levels 
of participation and governance, 

-- an open, public, continuing process, of which 
this DISCUSSION SYNTHESIS is an interim product; 

and it is NOT -- a voting or tabulation of preferences schema, 

— a decision process. 

s PUBLIC DISCUSSION DRAFT SYNTHESIS is -- an open invitation to you to 

participate in, and help shape, 
the redesign dialogue. 

"The collection and dissemination of education statistics have been a 
federal responsibility for almost 120 years. The federal role in 
education has changed considerably since then and the demands made 
for the scope, depth, and accuracy of information have increased 
accordingly. While the appropriateness of the federal government's 
role in the collection and provision of statistical information is 
not seriously questioned, debate about what and how data should be 
collected is recurrent. The reason for this is that statistics are 
not ends in themselves — they support and facilitate the attainment 
of other purposes. Many of these purposes are controversial and 
involve real or potential conflict not only within the federal 
government but among different levels of government" (Hawley). 


year 1905 has become a year for the revitalization of the "recurrent 
te." By reading this synthesis, you are participating in that debate. We 
ite you to continue your participation into other stages of this debate. 

tever your purpose for reading this document, we invite you to read it with 
se that you want to share with NCES your concerns and suggestions on 
cation data needs and improvements and, through subsequent products of this 
ject, with the education community at large. 

REDESIGN PROJECT is a broad, public, open outreach to the education 
□unity to suggest ideas for improving the adequacy, scope, content, accuracy 
iability, and usefulness of the Nation's data for education policy, 
inistration, and research purposes. The Project is concerned with objective 
specific recommendations and suggestions. 


In three public participation stages of the Project, the education community i 
being invited to 


"join the discussion of how a redesign of NCES‘ data programs 
will enhance opportunities to collect productive information 
and, further, promote innovative strategies for distribution, 
use, and analysis" (Cronin). 


STAGE 1, COMMENTS: To initiate the dialogue, we invited a number of 
individuals, organizations. Federal Government Departments and Agencies, and 
operating components of the Department of Education to prepare a paper follow: 
the guidance given in Appendix B. 

The invited individual authors were selected admittedly somewhat arbitrarily < 
certainly not systematically. They were people likely to (a) use quantitative 
approaches in their analyses, and thus be data users, and (b) have the intere* 
and time to produce a "thought paper,” not a research paper. We wanted a 
diversity of ideas to stimulate a dialogue ; there was no intent to represent 
known positions or to provide balanced representation across sectors of the 
education community. 

The organizations were selected because of an identification with education 
issues and concerns, and were invited to participate in any or all stages of 
the project, including the option of writing a paper. 

To facilitate public comment in response to, or as stimulated by, the initial 
set of papers, NCES asked a team of writers to distill the essence of the 
papers. The product of that effort is this PUBLIC DISCUSSION DRAFT SYNTHESIS 
Its purpose is to invite you to provide NCES with any major issues, comments, 
concerns, suggestions, and criticisms that you feel have either not been rais 
or have been insufficiently developed. 

November 8 is the cut-off date for incorporation of your comments into an 
expanded version of the SYNTHESIS to be distributed as part of the call for t 
Stage 3 Public Discussions. We want to hear from you; but we cannot promise 
that responses received after November 8 will have visibility beyond NCES, 

STAGE 2, ALTERNATIVES: Using tile PU8HC DISCUSSION DRAFT SYNTHESIS, and the 
invited papers, a team of data system experts is drafting alternative data 
acquisition eysteras. These could provide NCES, and the education community, 
with a rationale and a framework for acquiring a substantial portion of the d 
domains, or elements, about public and private elementary secondary schools, 
teachers, and students. 

STAGE 3, PUBLIC DISCUSSION: A series of public discussions will be held this 
school year around the Nation to focus on how the concerns of data users and 
providers can be addressed by proposed alternative data acquisition systems. 


participate in the Project, send your COMMENTS to NCES at the address below, 
participate in the Public Forums, request detailed information and background 
.erials to be available in late fall. 

Leslie J. Silverman 
Deputy Assistant Administrator 
Division of Statistical Services 
Phone: 202/254-5530 

Richard C. Taeuber 
Research Director 

Division of Elementary and Secondary Education Statistics 
Phone: 202/254-5530 

ling address: 

National Center for Education Statistics 
U. S. Department of Education 
1200 Nineteenth Street NW. 

Washington, D.C. 20208-1401 


The Synthesis 




liASIC DATA — DESCRIPTIVE STATISTICS 


Richard C. Taeuber* 


Basic data are the core of data items which NCES collects on a regular basi: 
provide descriptive statistics on the state of education in the Nation, Th< 
authors of the invited papers have quite different views of the purpose of • 
core and hence the content. Some authors want discrete, descriptive data i 
acquired from State administrative records. Other authors want the core to 
include sets of analytic variables to support continuing policy analyses. 

The Common Core of Data (CCD) refers to the discrete administrative data it< 
acquired by the NCES Elementary and Secondary Education Statistics Division 
the states who are summarizing Local Education Agency (LEA) data. The Nati< 
Education Association (NEA) refers to the CCD in the following way: 

"The Core represents the most basic data series within the NCES. I 
enables assessments of what was, what is, and what will be in a 
statistical sense. Annual updates to Core surveys provide basic 
statistical information on public schools, their pupils, personnel, 
finances ... the Common Core may represent the most heavily used sej 
of public school statistics. The Core is the cornerstone of 
educational information in the United States. No other public or 
private institution collects and maintains public education data to 
extent that NCES does via the Core." 


CCD items are simple, factual data that are "indispensible in compiling a 
portrait of the education system," (B. Turnbull) They are data on the 
characteristics of students, teachers, schools, and school systems that giv< 
educators and the public a general sense of education developments at the 
National and State levels. They are the data reported in The Condition of 
Education and the Digest of Education Statistics , the primary publications 1 
NCES, as well as special reports. 

A listing of "basic" data items is provided the REDESIGN by Vance Grant who 
many years, has headed the NCES Statistical Information Office. He estimat 
that he has talked with some 87,000 inquirers over the course of 29 years 1 
statistical information function. His paper offers the following list of m 
frequently requested data items: 


*Dr. Taeuber is Research Director, Division of Elementary and Secondary 
Education Statistics, National Center for Education Statistics, and Co-Dir 
of the REDESIGN Project. 


July 1985 List of Basic Statistics Frequently Requested 
the Statistical Information Office 
(The date of the latest published and/or readily avail* 
NCES statistics follows each item) 


Public Elementary and Secondary Schools 
Pupils 

Enrollment by grade* (Fall 1983 ) 

Enrollment by level (elementary vs. secondary)* (Fall 1978) 
Enrollment by age, race, and sex (Census data) 

Offerings and enrollments in high school subjects* (1981-82) 
Average daily attendance and average daily membership* (1980—; 
Average length of school year and days attended per pupil enr< 
(1980-81 ) 

Pupils transported at public expense ( 1980-81 ) 

Employees 

Classroom teachers by level* (1980-81) 

Classroom teachers by sex* (1980-81) 

Classroom teachers by teaching field (1979-80) 

Other professional staff by type of position and by sex* (by 
position only. Fall 1981) 

Nonprofessional staff (Fall 1981) 

Schools 

Oy level* (1982-83) 

Oy grade span (1982-83) 

School Districts 

By size of enrollment* (Fall 1981 ) 

Operating vs. nonoperating (Fall 1982) 

High school graduates 
Gy sex* (1980-81) 

By type of program (Spring 1980 senior class) 

Revenue receipts 

Frcm Federal Government* (1982-83 ) 

From State governments* (1982-83) 

Fran local governments* (1902-83, including other sources) 
From other sources (gifts and tuition and transportation fees 
(1967-68) 

Nonrevenue receipts (1980-81) 


Expenditure s 

Current expenditures for regular school program* (1932-83) 
Instruction* ( 1930-81) 

Salaries of classroom teachers* (1981-82 estimates) 

Salaries of other instructional staff* ( 1975-76 data for 
total instructional staff) 

Salaries of nonprofessional staff (1975-76) 

Free textbooks (1975-76) 

School library books (1975-76) 

Supplies and other instructional expenses (1975-76) 
Administration* (1900-81) 

Operation and maintenance of plant* (1900-81) 

Fixed charges* (1980-01) 

Other school services* (1980-81) 

Transportation of public school pupils ( 1980-81) 

Health and attendance services (1980-81) 

Food and other services (1980-81) 

Other current expenditures (summer schools, community services)* 

( 1980-01) 

Capital outlay* (1980-81) 

Interest on school debt* (1980-81) 

rlvate elementary and secondary schools 

Pupils 

Enrollment by grade (Fall 1978) 

Enrollment by level* (1970-71) 

Enrollment by age, race, and sex (Census data) 

Employees 

Classroom teachers by level* (1970-71) 

Other professional staff (Requested in Fall 1978; not readily 
available) 

Nonprofessional staff (Requested in Fall 1970; not readily available) 
Schools by level* (1980-81) 

High school graduates by sex* (1964-65) 


While all the items on this list are judged tc be important, those marked 
with an asterisk are considered critical items if we are to continue to 
provide adequate service to the public. 



Few authors responded with comments on discrete data items, even though the 
questions provided the authors (see appsndix B) did invite such comments. 
Grant's listing is based on the public's inquiries to NCES. The only other 
submitted list, provided by the National Education Association (NEA), sugges 
modifications to the current CCD program. (A listing of the items currently 
requested, as of spring 1985, in each CCD Part is given in appendix C.) 


CCD Component _ NEA Suggestions __ 

1. Public School Universe o Add spring membership. 

o Add full-time-equivalent classroom teachers 
by sex and eleraentary/secondary level. 

2. Local Education Agency 

Universe o No additions or changes. 

3. Local Education Agency 

Nonfiscal Report o Add fall membership by grade. 

o Add number of full-time-equivalent LEA 
employees in all employee categories. 

o Add number of full-time-equivalent teachers 
by individual grade. 

o Add presence or absence of collective 
bargaining agreements for teacher, 
administrator, and educational support 
personnel groups. 

4. Public School District 

Finance Report o Provide revenue by source consistent with 

NCES handbook, on financial accounting. 

o Provide expenditure by function consistent 
with NCES handbook on financial accounting. 

o Provide other uses of funds by category 
consistent with NCES handbook on financial 
accounting. 

o Provide special exhibits by category 
consistent with NCES handbook on 
financial accounting. 


v-v-u component 


NbA bugge3tions 


5. State Aggregate Nonfiscal 
Report o 


Add fall membership by individual grade. 


o Add full-time-equivalent employees by 
major assignment category, by State. 


o Add number of high school graduates. 

6. State Aggregate Fiscal 

Report o Make revenue, expenditure, other uses, an 

special exhibits detail consistent with 
revisions suggested for district finance 
data. 


o Add average daily attendance. 

o Add State law defining average daily 
attendance. 


o Add State aid formulae. 


UNIVERSE DATA 

The Common Core of Data is the NCES vehicle to provide basic information 
universe of public schools and school districts in the United States. Ha 
states: 

"Universe maintenance ... should provide sufficient data on schools 
school districts. While it is essential to have these universe list; 
to keep them up to date, the data there must be readily available to 
Department employees and other researchers on a timely basis." 


As Plisko states: 

"... the NCES universe file provides the location and enrollments of 
(public) elementary and secondary schools. The 1902-83 school year 
most recent year available. Yet private firms ... have put together 
universe files for 1984-85 school year that provide much greater 
information on each school building." 



Timeliness aside, tnere are multiple suggestions as to items wnicn could oe 
added to the NCES annual universe descriptor update programs; 

o "... the number of noninstructional personnel... on a district basis in 
terms of full-time equivalent units ... valuable to know what the ratio 
students to these different personnel is in each district ... essential 
assess accurately the number and types of non instructional personnel who 
provide services to students" (American Association for Counseling and 
Development). 

o "... respondents are allowed to designate the beginning of the range of 
the educational program as pre-kindergarten or kindergarten. It would 
provide more useful data ... to have designations related to the age of 
the child" (Natriello). 

o "In order to understand the extent to which the public schools are 

involved (even without providing financial support) with a variety of ne 
educational services such as day-care for young children or extended day 
programs for latch-key children, it would be important to request 
information on programs affiliated with each public school, even those 
which simply use the physical facilities" (Natriello). 

On the matter of detailed data on students, there are several suggestions to 
acquire enrollraent/raembership by grade, rather than grouping 1-12 as one itec 
As to other student descriptors: age, sex, and racial/ethnic groupings, the 
suggestions include: 

o "... big cities or urban areas will become even, greater proportionally 
minority and poor .,. inaccurate data on this population ... lead to 
policy decisions which address problems which no longer exist, problems 
which are not adequately defined and, in too many cases, problems that 
never existed" (Eubanks). 

o "Collect and report all student data so as to permit race by sex analyst 
to be performed" (Hilliard). 

o "Total FTE, ADA and ADM (although an FTE figure would suffice) and by 
elementary, middle and secondary level .. . by ethnic background ... and 
special programs — compensatory education, bilingual education, specia 
education, gifted and talented*’ (Qdden). 


Many authors advocate moving away from aggregation. They want the detail, t 
"building blocks," so that the information can be processed and aggregated a 
appropriate to the decision or inquiry at hand. 


"The annual figures should be published in the kind of report we (NCI 
used to call Fall Statistics of Public Elementary and Secondary Day 
j?choois. In preparing this report, we should emphasize speed rather 
precision, so that the data can be published before the end of the st 
year to which they relate" (Grant). 


FINANCE DATA 


The other major sector of the NCES basic data program is data on revenues 
expenditures of local school districts. Although the National Governors' 
Association (NGA) states that "the key issue in improving school administi 
tion is school finance," Barro states that "NCES currently produces what r 
fairly be described as skeletal information on school finance." The plea, 
with the universe data, is that without detail and disaggregation, the use 
left with relatively few answers and but little useful information. 

The suggestion is advanced that a candidate area for deletion from the CCE 
program is "Common core collection of annual school district finances that 
detail financial and revenue statistics for 16,000 school districts" (Pile 
(Readers should be informed that this is one of a very few recommendations 
deletions frcm the current NCES data program.) 

The counter argument for retaining and expanding the collection of finance 
at the LEA level is offered by Barro; 

"There are no NCES publications describing the distributions of rever 
or expenditures among local school districts, either nationally or wi 
States, even though such distributions (e.g., intrastate disparities) 
long been the central concern of school finance policymakers and 
researchers." 


A suggestion for sample data acquisition from the LEA's is offered by Odde 


"NCES should continue to collect State aggregate revenue and expendit 
data, and revenue and expenditure data for a representative sample of 
districts, where the sample is representative of each of the fifty St 
as well as the nation as a whole." 


Barro and Odden offer detailed suggestions on needed finance data items, 
primarily from a concern that the move toward but a very few aggregate items h 
significantly reduced the usefulness and utility of the data. For example: 

"Two reasons for the limited usefulness of current data are that 
expenditure data are not collected in sufficient detail to be connected 
with resource categories, and expenditure and resource categories are not 
coordinated. Consequently, information on dollar outlays cannot be linke 
to anything real" (Barro). 


Odden wants "expenditures by function, ... expenditures by program, ... 
expenditures for the general fund, restricted fund and capital fund ... (and) 
for revenues ... more subcategories of State, as well as local revenue." And 
Barro goes on to state that, "Specifically, I envision a system of combined 
expenditure and resource accounts, in which outlays are explicitly linked to 
resource quantities and prices." 


If Barro's and Odden's visions are realized, the resulting data set would be i 
basis for a highly analytic finance database and likely to include many more 
discrete data elements than the 40 acquired by the 1985 CCD, or the additiona; 
suggestions by NEA, or the 23 items on Grant's list. 


PUBLIC AND PRIVATE SCHOOL DATA 

Although these surveys are not a part of the CCD program, several authors 
express a view that private school data must be made part of the NCES core da 
program. The NGA states that the current Public School Survey and the Privat 
School Survey "should be among NCES priorities" and several authors call for 
two to be conducted in the same cycle so that the data collected in each can 
cross-referenced to the other. 

There are also calls for the Public and the Private School Surveys to be 
expanded so that the data can be more State relevant. A specific element of 
suggestions is an interest in more information on the private schools and an 
ability to compare/contrast them to public education. Hilliard states: 

"There is a need to be able to identify such wide variations in treatmen 
among private schools. Clearly, children vary in terms of the quality o 
educational experiences to which they have been exposed. By collecting 
more complete data from private schools , more extensive analyses will 
become possible." 


is on to recommend that, "As much as possible, collect the same data from 
.e schools as from public schools." 


,ds that, "Of interest in the future will be financing issues of public 
private schools. Basic finance information concerning both sectors 
I continue to be considered a core data element in any elementary/secondary 
;ion data system. " 


QUALITY AND USEFULNESS 

uality across all acquisition programs is an issue addressed in most, if 
.1, of the papers. David makes the most forceful statement: 

If the data continue to be as inaccurate in the future as they have in 
.he past, all other issues are moot. Careful choices about what data to 
iollect and clear reporting and interpretation cannot compensate for 
.naccurate data." 


nbull requests that, "... the highest priority be placed on strengthening 
•edibility of the data through such means as quality control and 
itency in data series." 

irther states that, "The value of demographic and other descriptive data 
ises greatly when the data are collected and summarized in a consistent way 
ear to year. When a measure changes ... trends ... can be exaggerated or 

i. » 


>r, she adds a word of caution: 

'... even when there seems to be good reason to tinker ... the potential 
.mprovement should be carefully weighted against the loss of 
:omparability. Stability in measures should win out in most cases." 

echoes this latter concern by stating, "Great care should be exercised to 
tat the figures are comparable from State to State and consistent from one 
:o the next. " 

isly NCES has a primary responsibility for quality, but Bishop states: 

'The central goal of the common core data program should instead be to 
>rovide high quality data that are comparable across States. This implies 
;hat NCES needs to work cooperatively with the States to insure that data 
reported are consistent with the common defintions that have been 
idopted." 


"Special attention must be given to holding the States accountable for- 
providing the Department with consistent and accurate data" (Plisko). 

"The overall objective should be for SEAs to recognize that they are 
participating in a process that is centrally controlled in order to meet 
national information needs in a technically defensible way" (B. 

Turnbull). 


’he PURPOSE of a program of basic or administrative data is addressed: 

"The primary purpose of the common core data program is, therefore, not to 
help the federal government make better decisions but to help the 
citizenry and political leadership of individual States monitor the 
progress and achievements of the educational system in that State. 

Progress and achievements can be defined only when some standard of 
comparison is available" (Bishop). 

"But as the principal Federal information agency on education matters, 

NCES has both a responsibility and an opportunity to serve and protect the 
interests of consumers of educational services. This is the spirit of the 
large state-by-state comparison chart of educational inputs and outputs 
which NCES has just published. That spirit should be present in a much 
broader set of services, such as those described above, which can 
stimulate, encourage, and generate pressure for the opening up of 
information about school functioning to perents and community. Such 
services were less important when schools were closer to their 
canmunities, and when there were strong parental communities coterminous 
with communities of children and youth. But they are important now, and 
will become increasingly so in the future" (Coleman). 


tn these calls for change and addition, the final words belong to Cronin: "But 
hese questions should be asked: Who needs it? Who will use it? For what 

purposes? These are deliberately hard questions, and they should be raised 
repeatedly about the entire program of data collection." 


THE NEED FOR PROCESS DATA 


Ward S. Mason* 


INTRODUCTION 


One of the most consistent themes in the papers is the need for process data. 
Historically several types of input data have been available, and in more rec€ 
years information on outcomes has been expanding. But surprisingly little has 
been known at the national level about what actually goes on in schools. The 
High School and Beyond Study is exceptional in providing such data, but only t 
two points in time. In economic terms, the question is, what kinds of schooli 
are our education dollars buying? <Plisko). 

Recent research, notably that associated with the effective schools movement, 
has shown that variations in school practice are associated with variations ir 
student performance <Hersh, NEA). The design of process measures must be bas< 
on research that identifies the variables most strongly related to student 
outcomes (Plisko). In turn, claims are made that the inclusion of process 
measures in the statistical system design will contribute to knowledge of how 
the educational system works. 

Selection of the key process variables is thus a crucial task. Here, as 
elsewhere, the selection needs to be theory-driven and policy-focused (Walberc 
Buccino, Hawley, Bishop, McClure, Selden). For example, B. Turnbull suggests 
that, "Ideally, the collection of data should be driven by a framework of 
questions that the data will be used to answer. Working backwards from intern 
uses through projected analyses to the specification of the data elements and 
methods of collecting them would result in an efficient and practical program. 

Following the lead of many of the paper writers, we have used a crude 
input/process/output schema to organize the synthesis. Student outcomes are tl 
"bottom line" of the system, but much policy interest focuses on identifying 
those inputs and processes that influence those outcomes. However, a couple o: 
points need to be made about the use of such a model. 

First, often there is no apparent consensus on the classification of variablei 
(for example, classifying teacher competence as an input or as a process). 
Second, almost any variable in the total system can be viewed as a dependent 


♦Ward S. Mason is an independent consultant in Potomac, Maryland. He was 
formerly a senior research associate at the National Institute of Education. 



variable for purposes of analysis or as a focus for policy intervention. 9 
the model may imply an inappropriate linearity. In an on-going dynamic prc 
the same variable, for example, student self-discipline, can be viewed as & 
input, process, or outcome variable, depending on the question being addres 
Fourth, studies of school and teaching effectiveness can be no better than 
selection and measurement of the proper student outcomes (Murnane). So use 
the model is for convenience in organizing the synthesis only, and the 
classification of variables is not to be taken as fixed. 

Since we have no established set of process indicators, it will be necessai 
many cases to undertake considerable effort to develop and test the measure 
selected variables. Further, judgments will have to be made concerning whi 
variables are appropriate for a national data system, which for State and j 
data systems, and which are more appropriate for special research studies. 

Process variables can be grouped in five major categories as follows: 

o Staffing 
o Curriculum 
o Instruction 

o The social context of instruction 
o Access to and use of external resources 

Some of these groupings and variables are more appropriate to the school le 
of analysis, while others to the classroom or other levels. Some writers, 
Hersh and Barro, argue that the school is the most important level of analy 
and certainly this has been the tradition of school effectiveness research. 
However, Walberg disagrees, arguing that, in his productivity model, school 
variables have attenuated effects and tend to be mediated by other variable 

A large proportion of recommendations in the reform reports of the past 2 y 
and of the improvement initiatives actually undertaken have been aimed at c 
more of these process domains. One recommendation is to track the nature c 
school improvement efforts being launched at the national, State, and local 
levels, and the trends in these sfforts over time (Selden). Note that it is 
possible to monitor reforms involving process variables at three different 
levels: 


o what process reforms have been adopted? 

o Have the process reforms been implemented? 

o What has been the effect of implemented process reforms? 


STAFFING 

Teachers. There is considerable consensus on the importance of data on tea 
and teaching. Of the top ten issues cited by Governors in their State of t 
State messages, the top two were teacher issues (National Governors' 
Association). 


Teaching is the factor most immediately related to student performance. But t 
attempt to identify the elements of teaching excellence and the factors which 
turn influence teaching excellence, lead quickly to a very complex set of 
considerations. Figure 1 represents one way of mapping these variables into a 
manageable number of sets. Each set is both an important focus of analysis an 
policy interest in its own right and a set of factors influencing student 
performance. 

The Adequacy and Quality of the Teacher Workforce . Many writers suggest the 
need for data on the teacher workforce that would pormit studies of supply and 
demand, recruitment, and retention. Both numbers and quality are of concern, 

Berryman points out that the nation's public elementary and secondary schools 
employed over 2 million classroom teachers in 1982-83. This labor force is 
comparable in size to the nation's military active duty enlisted force and 
officer corps, and is the largest professional and technical occupation. 

When school enrollments were declining, the supply of teachers was not seen as 
major problem. More recently, major issues in the larger society—namely the 
economic productivity of the United States, our competitive position in world 
trade, and national defense concerns—have called into question whether our 
schools are doing an adequate job in math and science education. Study of this 
question quickly reveals a shortage of qualified math and science teachers. 
Further, demographic data indicate that early elementary school enrollments ha 
started to rise again, while a large proportion of the present teacher work 
force is expected to retire or otherwise leave teaching in the next decade. 

Thus, the supply and demand for teachers has re-emerged as a general issue. V 
we lack the fundamental information required to predict supply and demand 
<Berryman, Barro). The Common Core of Data (CCD) collected by NCES is limited 
to gross counts of full-time-equivalent staff by level, State, and school 
district. No data on salary are collected, although the Digest reprints data c 
salaries collected by the National Education Association. 

Recently, steps have been taken to collect more complete teacher data. A sampl 
survey of individuals obtains information on teaching experience, training, 
assignments, work hours, compensation, and certain personal characteristics, 
a special "Study of Teacher Demand and Shortage," LEA's and other educational 
institutions are asked to report on teaching positions, vacancies, new hires, 
certification status, and teaching assignments (but not on teacher 
characteristics or compensation). Thus, considerably more information will be 
available. Unfortunately the individual data cannot be related to the district 
data, and neither can be reported on a comparative inter jurisdictional basis 
(Barro). 

Better data would permit attention to a number of important issues, including 
"how the teaching force is and has been changing with respect to certain 
quality-related attributes; how quality-related characteristics of teachers vs 
among States, school districts, and schools; how such characteristics relate t 
teacher compensation, and other conditions of teaching, and the state of the 
teacher market; whether teachers with different characteristics tend to be 
assigned to different types of schools and pupils; and how teacher attributes 
relate to pupil achievement and other measures of educational outcome. Without 


Figure 1. Facets of Teaching Quality 
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one can do Little more than speculate about such concerns" (Barro). 


Improvement in supply and demand studies will require careful attention to a 
number of factors: 

o Recent innovations in terms of career ladders, 
differentiated staffing, merit pay, etc., will 
require new classifications of teaching positions, 
o Turnover rates need to be calculated on an 
age-specific basis (Berry/NSF). 
o Matching of qualifications with assignments (e.g., is a 
math class being taught by a teacher with 
certification in English?) is needed (NSF). 
o Separate data on new hires will act as a set of 
"leading indicators", giving an early indication of 
change in the teaching force. 

Compensation is considered a topic of special importance <Barro, Plisko). T< 
compensation needs to be broken into subcategories like salary, deferred 
compensation, and other fringe benefits. Different periods of employment (9' 
11-month employment) need to be accounted for. Average salaries for broad 
classes of teachers mask important variations; it is necessary relate 
differences in salary to a broad array of personal characteristics and 
educational assignments. 

At the same time, district data are needed on the structure of salary schedu. 
and how teachers are distributed on them. Many reform recommendations focus 
merit pay, career ladders, and other forms of teacher incentives. We simply 
need to know more about "how teachers are paid in different places, how pay 
systems are changing, and the consequences thereof for educational costs, th< 
make-up of the teaching force, and ultimately the quality of teaching and 
educational outcomes" (Barro). 

Since many of the factors regarding the salaries and working conditions of 
teachers are set by State policy, teacher labor markets tend to operate with 
States. When sample surveys are used, State representative samples of teache 
are needed to facilitate comparison among states based on these State policy 
differences (Harrison, Barro). 

Many writers express concern that teachers often are drawn from lower abilit 
groups as measured, for example, by SAT scores (Murnane). Indeed, "the decl 
in SAT scores for new teachers is greater than that for the total population 
SAT test-takers, but we do not know the relationship between score declines 
teaching performance" (Berryman). 

The social origins of teachers in terms of socioeconomic and equity categori 
are also of interest. Many of the policy initiatives in the teacher area ar 
aimed at recruiting and retaining more able individuals. However, it is not 
entirely clear that we know how to identify and measure those socioeconomic 
personal characteristics that are most predictive of teacher competence or 
sus tained careers. 


of the long term supply of teachers--such as career intentions of high school 
seniorsi reasons they cite for not going into teaching, and public perceptions 
of the status of the teaching profes9ion-**would be useful (Selden). 

Teacher Preparation . Teacher preparation has long been a controversial topic. 

A number of major reforms in pre-service programs have been proposed. Several 
writers express the need to obtain data on at least some of the major dimension 
of that preparation, such as type of postsecondary institution and academic 
preparation (education methods courses vs. subject specialization (Hilliard, 
Selden); and type of program — undergraduate vs. graduate (Scott-Jones). 

Selden is concerned with the prevalence or status of different approaches to 
teacher preparation and the relationship between alternative approaches and the 
relative proficiency of teachers in the classroom. 

Certification . The characteristics of the individuals recruited, plus the type 
and quality of the preparation program, together define indicators of the 
qualifications of new teachers. Certification standards of States and the 
recruiting criteria of school districts represent attempts to evaluate those 
qualifications and screen entrants to the profession. Data on these 
requirements would improve our understanding of movement into teaching and 
mobility among districts. 

Conditions of Work . There are many working conditions that are felt to have 
important influences on job satisfaction and decisions to remain in or leave 
teaching (Rosenholtz). Conditions of work also have an effect on teaching 
behavior and therefore teaching effectiveness. Among those conditions cataloged 
by our writers are the following; 


o Incentives: salaries, salaries relative to 
the salaries of other occupations (Murnane); 
benefit packages (Berryman), psychic rewards 
(Rosenholtz), etc. 

o Class size and teaching load (Lehnen). 
o Opportunity for increasing professional 
competence (Hawley) 

o Opportunity to interact professionally with 
peers (Hawley). 

o Participation in school decision making 
(Rosenheltz). 

o Supervision/evaluation/support from school 

principal (Hawley). 
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paper-ana-pencix quarrying examinations, ana compliance witn certification 
standards, but not from direct measures of the ability of teachers to teach 
students" (Selden). 

State programs for testing teachers have been quite controversial and generall 
focus on subject-matter knowledge. But it also seems desirable to be able tc 
measure pedagogical knowledge, skill, and method (Selden), relationships with 
parents (Coleman), etc. More specifically, it is proposed to measure: 

o Ability to plan and conduct a lesson (Selden). 
o Setting and enforcing clear expectations 
for student behavior (Rosenholtz). 
o Imposing order and discipline (Hersh, Selden). 
o Teacher caring (Hersh); affective attributes 
(Hardeman). 

o Use of a variety of teaching strategies to deal 
with individual students (Hersh). 

The need for methods to evaluate non-traditional teaching methods is noted by 
Hardeman, while Natriello calls for improvements in performance evaluation. 

Teacher Attitudes . A variety of teacher attitudes, values and motivations 
(Hawley, Thomas) may influence student behavior on the one hand and teacher 
commitment to the profession on the other. Rosenholtz in particular makes 
commitment the keystone variable in dealing with both issues. Commitment is 
impacted by teacher rewards, teacher certainty (regarding efficacy), 
opportunities for skill acquisition and development, teacher evaluation, 
buffering (of the teacher from interference), teacher isolation, faculty 
collaboration, participation in decisionmaking, and organizational rigidity ar 
flexibility. She notes that some recent educational reforms such as minimum 
competency testing and career ladders have had unintended and sometimes negatJ 
influences on teacher commitment. Lehnen also focuses attention on teacher 
attitudes and sees teacher absenteeism as an indicator of burnout. For Hersh, 
the feeling of efficacy is a key factor. Both Lehnen and Thomas propose an 
annual survey of teacher personnel to monitor key variables. 

The Teaching Profession . In addition to viewing the body of teachers as a 
workforce, one can see it as a profession. Here one is concerned with issues 
such as how the profession is organized, how it is involved in labor relation* 
issues (Lehnen), the development of norms governing behavior, the establishmei 
of standards, and the restructuring of rolee and careers (Usdan). The public'* 
perception of the status of teaching (Seldon, Scott-Jones) is relevant here. 
Understanding patterns of teacher mobility, both among teaching positions and 
into and out of the teaching force, is important (Murnane). 

Conclusion Regarding Teachers . As suggested at the beginning of this section 
and diagrammed in Figure 1, the issues regarding collection of teacher data a: 
quite complex. Good teaching is often identified as the key to good educatioi 
We need better data on the adequacy and quality of the teacher force, and we 
need to know how to identify teaching excellence* However, our research 
knowledge base is not yet robust, and improvement of data systems for teacher; 
will have to proceed hand-in-hand with research on these issues and the 
development of reliable measures. 


The selection of strategies for collecting data to address the varied issu 
purposes is also quite difficult. Barro addresses this problem: 


"If NCES does become involved in a msjor way in collection 
of data on teachers, it will have to make some strategic 
decisions at the outset. Among these, the most basic concern 
the choices of units of analysis, respondents, and level of 
detail. I consider here some of the diverse purposes for 
which teacher data might be wanted and the degrees to which 
these purposes might be served by different data collection 
modes. 

"One possible objective, clearly of current interest to tho 
Education Department, is to assemble state-by-state data on 
teachers to add to the comparative displays of state 
education statistics (the famous "wall charts") distributed 
by the Department this year and last. The only teacher data 
now included are pupil-teacher ratios. Other items of 
potential interest include statewide averages of teacher 
experience, training, and other characteristics and 
indicators of the level of teacher compensation in each 
state, such as salaries paid, on average, to teachers with 
specified standard characteristics. Such information could 
be obtained from state education agencies (which, in some 
cases, would have to institute new data collection procedures 
of their own to obtain the information from LEAs); from NCES 
censuses or, possibly, sample surveys of individual 
districts; or, in part, from state-representative sample 
surveys of individual teachers. 


"Another, much broader objective is to construct a general 
teacher data base that can be used to support a variety of 
research and policy inquiries. Such a profile should include 
information on teacher characteristics, teacher compensation, 
and the conditions of teaching. Disaggregation to the state 
level is the minimum required for such a file to be at all 
useful, and for most research purposes that level of detail 
would not suffice. For instance, it would be difficult to 
derive valid conclusions about teacher quality, patterns of 
compensation, or relationships between teacher 
characteristics and outcomes without distinguishing, at 

* m ° ng urban » suburban, and rural districts; districts 
of different sizes and districts of different levels of 
income or wealth. For in-depth analyses in any of these 
areas individual district data would be required. Such data 
could be obtained through state agencies or from LEAs 
directly. The choice between the two seems to hinge on (a) 
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set of teacher-related items to the Common Core of Data. 

''A somewhat more specialized reearch-oriented objective is 
to assemble the data needed to address teacher supply and 
demand issues, including the key issue of how teacher supply, 
and in particular its quality dimension, responds to changes 
in compensation and other market conditions. Some aspects of 
these issues, especially questions on the supply side, can be 
addressed through sample surveys of individual teachers or 
college graduates--provided, however, that the samples are 
drawn not merely to be nationally representative but to 
allow comparisons among states and types of districts. Other 
questions, including many on the demand side, require 
in-depth data from samples of school districts such as 
salary schedules and the distribution of teachers upon them. 
In particular, an analysis of the flow of persons into and 
out of teaching would seem to require district-level data, 
specifically including detailed information on those entering 
and leaving the teaching force. 

"Finally, a narrower, but currently high-priority objective 
is to assemble data sets suitable for evaluating the effects 
of the major changes in teacher pay systems and certification 
standards now being instituted around the country. This 
would probably require data from sample districts in states 
establishing the new systems (i.e., merit pay, career 
ladders, teacher proficiency examinations, etc.), with 
special emphasis on data concerning newly hired teachers and 
teacher turnover. It would also require collection of 
longitudinal data to determine the effects of the policy 
changes over time. 

"This list by no means exhausts the possibilities, but it 
suffices to make several points. First, geographically 
disaggregated teacher data are essential for research and 
policy uses. State-by-state data will serve some purposes, 
but for many research applications district-level data will 
be required. Second, national data, and hence surveys based 
only on nationally representative samples, are of very 
limited value. They provide general background information 
and good numbers to use in speeches but contribute little to 
understanding how the teacher system works. Third, whatever 
the unit of analysis and whatever data collection strategy is 
used, it is important that data on all the relevant aspects 
of teaching be collected together. That is, data on teacher 
characteristics, compensation, working conditions, etc. 
should all be collected from the same respondents at the same 
times, so that relationships among these variables can be 
explored." 


The papers say relatively little about data on principals, superintendents, 
other non-teaching staff beside endorsing the continuing collection of gross 
counts. However, the central role of the principal in establishing the prop 
climate has been noted in the school effectiveness literature and is discuss 
below in that connection. 

Elementary principals are more likely to assume the role of instructional 
leader, while secondary principals are more frequently seen in an adrainistrf 
or bureaucratic role (McPartland). 

The National Commission on Excellence in Education pointed to the need for 
school leaders and administrators to provide leadership in enhancing the 
involvement and support of parents, citizens, businesses, and others in the 
schools. Tracking their efforts in this arena would be desirable (Selden). 

Several writers express a concern that the size of administrative and suppoi 
staffs may have become disproportionate to the size of teaching staffs, and 
propose various measures to monitor this issue. Possible measures include: 

o Administrative staff vs. instructional staff 
(Hannaway). 

o Central office administrators ve. "those whose 
activities and locations are school-centered" 

(Harrison). 

This seems to exhibit a skepticism about the amount and kind of support that 
teachers need from school or district-level personnel. 

Other writers indicate an interest in the time allocations of principals 
(Hannaway) and their career paths (Hawley). The American Association of 
Counseling and Development notes the need for data on counseling personnel, 
including time and task analysis. 


CURRICULUM 

Under the heading of "curriculum" we address the basic questions about what 
taught in school. The issues are somewhat different for elementary and 
secondary schools. At the elementary level, where there is much more unifoi 
of content at the subject level, a major concern is the relative emphasis or 
"basics" (however defined) and higher order skills. At the secondary level, 
where the alternatives implicit in tracking and electives begin, the concerr 
the ability to describe in detail different configurations of the learning 
experience. 


Courses and Enrollments . Despite the seeming uniformity of elementary 
education, writers express a need to have more detail on the topics and text 
content for different subjects and grade levels (B. Turnbull, Berry/NSF, NSI 


o What subjects are covered at what level of difficulty 
(Hawley)? 

o How much time is devoted to different subjects and 
topics (Peterson)? 

o Is there a core curriculum that is well articulated 
across grade levels (Hawley)? 
o Is it a "tightly coupled" curriculum (Hersh)? 

One problem noted with minimum competency tests is their tendency to focus or 
basic skills to the exclusion of higher order cognitive skills (Murnane). Sj 
some teachers tend to teach to the test, the testing movement may have the 
effect of crowding out higher order thinking skills from the curriculum. This 
effect might be overcome if tests could include a balanced attention to highe 
order skills (Peterson). 

At the secondary level, the need for data on a wider variety of courses is 
recognized. A preliminary task is to develop a common nomenclature of course 
(Hilliard). Cronin is concerned that the "new basics" emphasized in Secretai 
Bell's report on excellence, A Nation at Risk , is still too narrow, consistir 
of the traditional academic subjects of reading, writing, mathematics, scienc 
and only one "new" secondary subject--computer studies. Any systematic 
evaluation of education in the United States should include art, music, and 
foreign languages. As with elementary curricula, we need to go beyond course 
titles, which are often inflated, to determine the topics actually covered. 

One important clue to course content involves the analysis of textbooks, 
chapters and topics actually used and assigned, etc. The question is whethei 
textbooks have become lees demanding, academically, over time, or have eimplj. 
become easier to read (Selden). 

The National Commission on Excellence was concerned that it could not make 
international comparisons on important curriculum issues (Selden): 

o Whether the courses taken by students in high school in 
this country differ from the courses taken by high 
school students in other countries, and 
o Whether, and how, the substance of the courses typically 
taken by students in high school or elementary school 
differs among industrialized countries. 

Buccino endorses the need for international comparisons but cautions about tl 
difficulties of interpretation. He calls for the collection of related data 
needed for valid interpretation. 

It is very important to look at the fit between secondary and postsecondary 
curricula (Reisner, Usdan). Accurate data on the content of elementary and 
secondary schooling are needed to enable an accurate analysis of the extent 
which students are receiving instruction in skills and subject areas needed 
successful transition into postsecondary programs. More specifically: 

o Where should the responsibility for remedial 
education lie? 

o Are college-level courses moving to the secondary level, 
and if so, how do college curricula need to be adjusted? 
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mathematics courses are essential to making jugdments on whether our schools 
doing an adequate job of preparing future citizens in these jreas. 

At the elementary level, "Experts presume that 100 percent of students 

enrolled In reading and mathematics in grades K-6. However, **grades 7-B) 
considerable variation, espeoialiy in the lower grades as well as grades B) 
in the proportions of students enrolled, at each grade level, in other course 
areas such as English/Language Arts, Spelling, Handwriting, Science, Health, 
Social Studies, Computer Sciences, Music, Art, Foreign Language, etc. 

(Reece/AAP). Estimates of enrollment tn each of the major course areas every 2 

years are proposed. 


Preparation of projections of total enrollment of all students in all courses 
for each level K-12 is endorsed by the publishing industry (AAP). 

The enrollment patterns of special needs groups are needed for policy purposes. 
For example, what kinds of handicapped children are being mainstreamed at what 
levels and what subjects (Cronin)? What are the patterns of participation in 
bilingual programs (Valdivleso)? 


Odden proposes bringing course and enrollment data together in a new way. For 
particular academic fields (e.g., science), numbers of student course sections 
could be adjusted to a standard such as number of sections per 1,000 students. 


NCES has obtained data on offerings and enrollments in high school subjects at 
infrequent intervals. The High School and Beyond study contained some data of 
this kind. According to those responsible for providing statistical informatio 
to users, such data have been particularly valuable and should be collected at 
more regular intervals (Grant). 


Some writers go beyond the traditional curriculum to note the need for data on 
opportunities for and participation in programs of community service and 
extracurricular activities (Cronin, McPartland, Coleman). Coleman views such 
activities ae a strategy for combatting the effects of weak or disorganized 
families, or weak and disorganized communities, by building compensating social 
structures. Hersh notes that opportunities for student responsibility are 
characteristic of effective schools. 


Requirements . Curricular requirements help to make up the culture of the 
school, its climate of norms and expectations (see section on the Social Contex 
of Instruction below). They are covered here as an additional dimension of 
curriculum description. 


Curriculum requirements are set initially by the State, but are often 
supplemented at the Local level. Comparison of requirements at the two loveL 
ould reveal interesting ranges of variation (Selden). As indicated above, t 
is also an area where international comparisons are thought to be important. 

equirements for high echooL graduation are an important sub-set of 
requirements, and proposals for raising these requirements have passed or are 
pending in a number of States. Other suggestions include: 

o A mapping of the similarities and differences in State 
requirements. 

o High school graduation requirements adjusted 
to some common norm across States (Odden). 

o Specification of requirements in terms of both specific 
courses or content and number of units (AAP). 


INSTRUCTION 

"Instruction" is a general category covering the methods or technology of 
teaching, grouped here under the headings of organizational arrangements, 
instructional practices, and school resources. 

Organizational Arrangements . Practices regarding tracking and grouping have 
been among the most controversial in education. Patterns for elementary and 
secondary education are somewhat different (McPartland). Elementary .schools 
tend to assign students to classes randomly but to sort students into 
homogeneous achievement groups within the classroom for instruction. In 
contrast, secondary schools tend to create more homogeneous instructional gri 
by placing students in programs and tracks (e.g., college preparatory, 
vocational, general) by achievement level. Within tracks students may be 
further separated by level, e.g., high, medium, and low sections of the same 
English course. Further, with respect to corrective instruction, elemontary 
schools are more likely to use within-class instruction while high schools m 
more likely to have separate remedial classes with specialized teachers. 

McPartland speculates that researchers would likely agree that such tracking 
grouping practices have negative effects on the development of students in t) 
lowest groups, but that teachers would be equally agreed that homogeneous 
grouping of students produces greater learning because instruction io target' 
to student needs. However, he feels that the appropriate experimental resear 
has not yet been done and that it is therefore too soon to draw scientific 
conclusions. In the meantime, we need to collect data on these practices so 
that they may be properly assessed. He notes in particular that no good dat 
are now available on grouping practices at the middle and junior high school 
levels. 

A variety of other dimensions of organizational arrangements are of interest 


o Class size with and without teacher aides (Hawley), 
o Size of school and how students are assigned to 
instructional units (Hawley), 
o Student assignment to various classes and special 
programs (i.e., special education and gifted and 
talented programs) (Thomas). 

Finally, various forms of team teaching and differentiated staffing represent 
organizational arrangements that need to be assessed (McPartland). 

Instructional Practices. 


Pedagogy. Potentially, there is a very wide variety of pedagogical practicei 
that can be employed by teachers, and a systematic classification of them wou. 
seem to be prerequisite to exploring this domain. Note that these factors ar< 
among those which are most clearly under the control of the teacher. At one 
level is the suggestion to document the mix of such practices as whole class 
teaching, small group instruction, programmed instruction, individualized 
learning, peer tutoring, open education, outcome-based education, mastery 
learning, and interactive teaching, classroom discussion, laboratory or 
activity-centered instruction (Hawley, B. Turnbull, Reisner). To be useful, 
such data presumably would have to be coupled with information on subject, gri 
level, types of students, etc. 

Other practices and dimensions of practice include: 

o Use of a variety of teaching strategies (Uersh). 
o Quality of instruction (McParland). 
o Appropriate level of instruction (McPartland) 
o Incentives (McPartland). 

o Diagnostic data on reading, comprehension, and higher 
order thinking skills (Eubanke). 


Time, Flowing out of the research on "time on task" is a broad interest in 
the structuring and use of time in schools. Many writers are interested in 
measuring academic learning time, with careful distinctions between time 
allocated and student engaged time (McDonough/AACD, Selden, Uersh, Walberg, 
Odden, Peterson, Scott-Jones). Time spent on homework is also considered 
important (Hersh, NSF, Selden). 

The parallel issue for teachers is hours per week taught by teachers (Hawley) 
The problem of interruptions, as for public address announcements, is also no - 
(B. Turnbull). 

At the macro level, there is an interest in the length of the school year and 

day (Scott-Jones, Peterson) and methods used to schedule periods during the d« 
(McPartland). 

School Resources . Schools generally have a variety of learning resources at 
hand. The availability and use of libraries, audio-visual equipment, program) 
instruction, media centers, computers, and other technology are of general 
interest (Buccino, Natriello, Hawley, Miller). However, one paper cites libra: 
data as an area where deletions might be feasible (Plisko). 


form reports. While acknowledging the NCES collection of library statistics, 
e believes such counts are of limited value if they cannot be related to data 
student usage, relation to curriculum, mode of organization and delivery, 

c. 


e use of computers and other high technology products is seen as an indicator 
the responsiveness of the schools to the computer age (Peterson). Computer 
teracy has beon identified as one of the "new basics" for modern education, 
mputere are also being used to deliver instruction. However, it is not enough 
have simple counts of computers available, or even how much exposure students 
ve to them; data are needed on how they are used. 

e National School Boards Association (NSBA) expresses an interest in other 
pects of electronic technology as well: 

o Use by school management to gather timely data about 
the success of the schooling enterprise; and 

o Impact of technology on the roles of school 
professionals, the diversity of instructional 
strategies, and the diversity of student roles. 


THE SOCIAL CONTEXT OF INSTRUCTION 

rriculura and instruction, covered in the previous sections, generally define 
at is happening at the classroom level and impact most directly on student 
rformance. However, a number of lines of inquiry in educational research have 
monstrated the importance of the school as a unit of analysis and focus for 
proveraent. This has been the tradition of school effectiveness studies, 
though McClure and Plank feel that the school site is a neglected level of 
ta collection. The context of the school has important influences on how 
assroora level variables operate. (However, note that Walberg tends to 
-emphasize the importance of these factors.) 

shop outlines the two major issues at the school level as follows: 

o "What is it about a school that causes students to 
learn more, to develop better character, or become 
more employable? Associations have been found between 
school climate and learning and lists of school 
effectiveness characteristics have been generated. The 
lists seem reasonable but the research that has 
generated them suffers from serious methodological 
weaknesses (Purkey and Smith, 1902), 

o "How are effective schools created? How does a school's 
ethos evolve? Since the school characteristics that 
are associated with school effectiveness are not under 
the direct control of a principal or school board 
we need to describe the relationship between 
administrative actions and school climate." 


These factors will be °is , relationships, and access to and use 

culture of the school, discipline, social relations i 

of external resources. 


, „ «. ,. h1a rateqorv receives little attention. The 
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elementary and secondary schools as well as tnc 

levels. 


The Culture o f the School . The culture of the school, or ethos, consists of t 
goals or standards, norms and expectations, regulations and policies, present 
a school (McPartland, NEA, Selden, Bishop). The principal often plays a key 
role in establishing these expectations. High expectations have been noted as 
characteristics of effective schools (Hersh), but the dynamics of how these 
expectations are established and how they function are not well understood. 


"Needed here are surveys of state, local and classroom 
practices in setting, communicating, and enforcing academic 
standards for students...We do not know: 


o what standards States and local districts apply to 
promote students from level to level 

o what criteria teachers apply in assigning grades and 
how they might be changing over time 

o enough about what parents expect of their children in 
school, or how well these expectations are communicated 
to students. 


In order to obtain this information, we would need 

new or expanded surveys of accreditation bodies, local school 

administrators, teachers, parents, and students*' (Selden). 

McPartland notes as important dimensions of the general climate of a school tl 
strength and direction of shared goals and expectations, and their clarity an' 
consistency for subgroups of students and teachers. 

Discipline . The degree to which students identify with a school and agree to 
guided by its norms has an important effect on the level of discipline in the 
school. The development of self-discipline is one of the goals of the school 
The level of incidence of discipline problems, victimization of students and 
teachers, and absenteeism are indicators of social disorganization. Problems 
measuring these factors are noted by Smith. Use of corporal punishment, 
expulsion, and suspension are forms of response to these problems (Scott-Jone 


Social Relationships . A number of kinds of social relationships are importan 
in defining the social context of learning. In terras of teacher-student 
relationships, McPartland points out that elementary schools tend to be 
"pupil-oriented" while secondary schools tend to be "subject-oriented". The 
middle school movement is viewed as an attempt to preserve something of the 
pupil oriententation of elementary schools while also helping the young 
adolescent move toward autonomy and self-direction. 

Teacher-to-teacher relationships are also important. Innovations like team 
teaching and differentiated staffing change the nature of these relationships 
considerably. How these changes influence student learning is still not well 
understood. 

Finally, the learning environment is influenced by the number and kinds of pe 
groups that form among students. 

"Which students an individual associates with may have a 
powerful effect on personal development. How many peer 
groups and close friends an individual is attached to may 
determine how influential any particular circle of friends 
may be. The overlaps among a student's associates in class, 
in extracurricular activities, and outside of school may 
also influence peer group effects...much more needs to be 
learned about how to coordinate the forces of the peer group 
to help students achieve academic and developmental goals" 

(McPartland). 


ACCESS TO AND USE OF EXTERNAL RESOURCES 

For many purposes, external resources might be better considered as inputs or 
context variables, but they are considered here for convenience and because i 
some ways they act as process variables. For example, a number of writers 
express interest in the availability and use of non-school educational resoui 
3uch as pre-3Chool programs (Hilliard, Thomas), extended care programs 
(Moser/Lutherans), after-hours schools (Murnane), volunteers (Hawley), museum 
and off-campus learning (NSDA), tutoring (Hilliard), etc. Consideration of tk 
resources draws attention to the fact that students learn in both school and 
non-school environments and that one cannot fully understand the contribution 
schools to learning without accounting for these other influences. 

Coleman gives the most detailed attention to the role of the family and 
community. He believes that school effectiveness is greatly influenced by tl 
presence or absence of strong and supportive family and community environment 
He proposes obtaining data on the social structure and its relation to the 
school, including measures of family characteristics, the school's relation 1 
the family, community organization, and the school’s relation to the communit 

Thomas also notes the importance of parental involvement, values, and 
familiarity with their children's schools. Both Thomas and Coleman call for 
surveys of parents. 


SPECIAL WfJiw 


A number 0 e writers raise issues concerning “ 

schools that cut across the process vanabLe categor 

• *. < nn i ft pant sub-set of schools for which special 

Big city schools constitute a significant sub set ol a f 

® . j i rwAciv fn better understand the problems ot 

atudiea and items are needed in order to oettei. uu . 

^citing the minority and poor (Eubanks, Usdan). Eubanks P-P-- that ^ 

and student data be analyzed in terms of social class fac *°”* “ij* 3 *° r 

data on the type and nature of communications that occur between the school and 

the community. 


At the same time, Barker (Rural Education Association) believes that small rural 
schools have special problems that require separate analyses. Uo suggests that 
educational data be reported by school district size as follows; 


less than 300 students 

300-999 

1000-2500 

more than 2500 

He further suggests that it is important to recognize that "rural" is a very 
diverse category, encompassing such situations as, "an island hamlet off the 
coast of Maine, an Alaskan native village near the Arctic Circle, a coal mining 
town in West Virginia, a ranching area in Wyoming, an impoverished community In 
the Mississippi Delta, a ski resort section of Vermont, or a prosperous grain 
farming region in Iowa...". 

Bilingual education can be expected to continue as a focus of policy interest 
for some years to come. Valdivieso details this problem as it effects the 
Hispanic population. He proposes that NCES develop a construct for "at risk" 
students that goes beyond proxy measures to those which directly measure those 
conditions that place students at risk. 


Cronin believes that federal and state policymakers need to know the answers to 
such questions as: 


o "How many bilingual students remain in school 
and graduate? 

o "How many graduate from transition bilingual education 
classes or programs? 

o "What is the level of linguistic competence, both in 
English and in another language, of those who 
participate in bilingual programs for one school 
year or more?" 


Scott-Jones notes difficulties in using tests of English proficiency to identif 
students needing special English services. She calls for data on the nature of 
the special instruction they receive > particularly whether an immersion or 
transitional strategy is used. Differences in student outcomes for students In 
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bilingual education. 

ocational education has had its own statistical system. The need for and 
problems in collecting data on enrollment by course of study are outlined by t 
National Center for Research in Vocational Education. They find that it is 
necessary to distinguish between those who take vocational courses as a route 
future employment and those who have other motivations. They also make a 
distinction between data collected primarily for policy purposes and that for 
labor market information. Most data for policy purposes can best be collected 
with special studies conducted on a sampling basis. Accurate data on program 
completers for the Occupational Information System require a census. 

he ability to analyze process data for various categories of special needs 
students such as the disadvantaged or handicapped is recommended (Hilliard, B, 
Turnbull). For example, Cronin is interested in the "numbers of handicapped 
students who have been served in programs (such as those funded by 94-142) and 
who have: 


a. been mainstreamed, sent to less restrictive alternatives, 

b. graduated into either vocational or college preparatory 
programs, 

c. become gainfully employed or enrolled in college." 

Suggestions are also made for collecting process data on preprimary schools (B 
Turnbull, W. Turnbull, Hilliard). This information is needed in order to deal 
with policy issues concerning the availability, cost, standards, and impact of 
these schools. 

There are suggestions that data should be collected on private schools. Such 
studies should use items on curriculum, instruction, social context, etc., tha 
ermit comparisons with public school data (Scott-Jones, Hilliard). "For prof 
private schools" is a special sub-set to which the private school survey Ghoul 
be extended (Murnane). A crucial need is for data that will sort out what, if 
any, self-selection effects explain differences between public and private 
schools (Hannaway). 


SUMMARY 

This review of the need for process data is in part a recitation of suggestion 
for national data on new kinds of variables. We have tried to indicate the 
special policy Issues which these variables are intended to illuminate. Many 
the suggestions are couched in terms of adding variables to specific studies 
such as High School and Beyond, while others are linked to the need for new 
surveys. This is not the place to go into the details of such design issues 
except to note that some kinds of policy analysis require bringing together 
different kinds of data from different studies. For example, understanding of 
teacher supply and demand issues requires both organizational data on staff, 
positions, vacancies, salaries, bsnefits, etc., but also individual teacher da 
on satisfaction, working conditions, motivation, etc. To facilitate this, it 
will be necessary to look at such matters as the use of common categories (o.c 
salary intervals) and overlapping samples (e.g., school data and teacher data 
from the same places). 


We began this chapter with the admonition that variable selection needed to 
theory-driven and policy-focused. In synthesizing many papers based on a 
variety of perspectives, we have inevitably done violence to that principle, 
is important, as the redesign proceeds, that the selections from among these 
many suggestions be based on some organized framework that makes the outcome 
more than a new laundry list. 


Christopher T. Cross* 
Betty Balcorab** 


THE NEED FOR OUTCOME DATA 

the more than 50 papers that have been submitted to NCES, a vast array of 
lily specific needs are addressed. Many of the respondents unabashedly 
ress their own particular needs. There is also a great deal of talk about 
overarching theuies of excellence and equity, the rallying cry of the 
rent reaction to A Nation at Risk , the report of the National Commission on 
lienee in Education. A number of writers discuss areas that can be readily 
sured numerically such as dollars spent, teachers and other staff employed, 
courses offered. But no matter what the theme of their paper, virtually 
ryone concedes, at least obliquely, that the primary mission of our schools 
to educate our children. 

schools must send our children out into the world as literate young adults 
can enter the labor force or continue their education toward that eventual 

l. Wo must not teach our children merely to take tests so we can measure 

m. We must teach them basic and higher order skills that will serve them 
oughout their lives. The acquisition of skills and ability to function an an 
It constitute the outcomes of education. The outcomes of education are what 

system is about; they are why we bother, they are our end product. In the 
ds of one of the respondents, "we need to know what students are... loa ruing 
(our) classrooms." (See Peterson; 3ee also Mawloy; McPartlarid.) 

need for and ability to collect outcome data presents, perhaps, the most 
vocative set of issues discussed in the papers. Several writers argue quite 
suasively that without outcome data policymakers at ail levels are deprived 
the information they require to make informed decisions about the 
ocations of resources, the improvement of practices, and the formulation 
enforcement of rules and regulations. As Hawley notes, "In particular, tho 
stent information tells us too little about the outcomes of education. And, 
n such information is available, measures that might account for differences 
ng students and school systems often are not.'* As Uuccino notes, we have 
n blessed with a great deal of information on input while output data has 
n "scarce and inaccurate." As stated, one of tho hallmarks of the curreut 
entlon being given to education is a shift from input data to output data. 


ristopher T. Cross is Executive Vice President and Chief Operating Officer of 
versity Research Corporation. Previously he served a 3 Republican Staff 
ector of the U.S. House of Representatives' Committee on Education and Labor. 

etty Balcomb is a research associate with University Research Corporation. 
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development of good documentation, available through a central source like 

NCES, about the comparability of the different data ases. 

In its paper, the National Education Association (NEA) notes that of the 
student outcome data currently available (reading rates, achievement rates, 
dropout rates), each is inadequate. NEA suggests the following major revisions 
in the collection of student outcome data: 


o Expand and standardize the definition of student performance outcomes 
to include more knowledge areas such as linguistic, musical, spatial, 
logical-mathematical, body-kinesthetic, and personal knowledge. 

o Develop measures to enlarge the view of student outcomes built upon 
the National Assessment of Educational Progress. 


o Collect fall and spring enrollment figures. 

o Standardize definitions of dropout, attendance, and literacy. 

o Convene an advisory group to study methods suitable for measuring 
dropout rates and student mobility. 

Buccino calls for the development of an organizational framework of five data 
series based on the 1975 work of Godea. Buccino's first category of data is 
outcomes. He devides this into two components—-tests and credentials. 

Although concerned about potential misuse of tests, Buccino acknowledges that 
tests provide a measure of what people know and of what intellectual and 
performance skills they possess. Beyond testing, Buccino urges that NCES 
collect data on the earning of credentials as another measure of education 
outcomes. Included would be such things as diplomas, certificates, and degrees 
Outcomes, he notes, should constitute the focus of any program to monitor the 
education system. Outcome data represent Information recognizable to the 
public and to noneducator decisionmakers. 


In his discussion of student achievement, Selden presents a list of ten output! 
that the Excellence Commission reported concerning student achievement: 


o A general pattern of decline in standardized achievement test scores. 


o A decline in Scholastic Aptitude Test scores. 

o A decline in college board achievement tests in such areas as physics 
and English. 


o A steady decl e 
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o Poor performance on NAEP tests of higher order skills, reading, 
writing, and math. 

o Poor U.S. performance on international achievement comparisons. 

o A decline in both the number and proportion of students scoring over 
650 on the SAT. 

o A perception that the most gifted are achieving below their aptitude. 

o A decline in Graduate Record Exam scores. 

o A higher number of functionally illiterate adults and youth. 

Selden also reports that the commission was unable to report on another seve 
areas of achievement because of the lack of data: 

o What students know about concept and principles such as the central 
theme of Moby Dick . 

o Current international comparison data. 

o Ability of U.S. students to solve complex problems by finding and 
interpreting information, applying analytic reason, and expressing 
conclusions effectively. 

o Trends over time in educational productivity. 

o Trends over time in basic achievement and functional literacy. 

o The range of average student achievement across significant political 
units. 

o Trends in achievement in states and districts facing different 
challenges. 


DEFINITION AND MEASUREMENT ISSUES 

As noted earlier in references to several papers and as evidenced by a 
continuing dialog between education leaders, policymakers, and the press, th 
is an abundance of discussion (some would say argument) about how to both 
define and measure outcomes. If the subject of outcomes is provocative, the 
question of how to measure outcomes has been the lightening rod in this fiel 

Cited by a number of writers are problems of comparability, standardization, 
relevance to the curricula, and the difficulty of measuring such higher orde 
skills as writing and conceptualization. 

In their paper, Plisko, Ginsberg, and Chaikind present a chart showing the 
categories covered by a number of elementary and secondary data bases. The 


International Association date base. 

r\f Frtnpfition data base that focuses on 
NAEP, in fact, is the on y epar National Institute of Education, 

outcomes and it is funded and managed y 

i <-vf a M^i-ional Bureau of Educational Standards, 

t-jaibpru who uraes the creation of a National 

argues for the establishment of absolute measures. He cites the 
John Carroll in noting that in 1925, L.L. Thurston had attempted to calibrate 
mental abilities and tasks to chronological age and learning time. If 
Thurston's work had been continued, we «l«ht be in a much better position today 
to actually have both absolute measures and a wide range of longitudinal 
information. Noting that the athletic world has the finest set of performance 
measures, VJalberg says of educational measurement, "It is as though each test 
publisher and teacher had a different meter stick; and yet there is no way to 

equate them." 


We need test scores. No one recommends their elimination. But we need to keep 
in mind the inherent shortcoming of test scores, particularly the lack of 
comparability. One author suggests that test scores be accompanied by 
descriptions of what was tested (B. Turnbull). 


One possible solution which is advanced is to calibrate teats with a national 
standard test, like NAEP. Although that would be limited to three age levels, 
it would be a step in the direction Walberg seeks. 


Another author (Harrison) noted that in the early 1970's, the U.S. Office of 
Education (OE) spent a great deal of time and money on the development of such 
an equating instrument, the Anchor Test by Dr. Charles Hammer. Regrettably, it 
was neither publicized nor used by OE. An appropriate area of inquiry at this 
juncture might be to re-examine the Anchor Test to determine if its reeurrectior 
is possible. Harrison warns against either a federal or State attempt to desigr 
a testing program to make outcome comparisons while urging that States agree on 
a set of achievement tests that could be administered by each State or the 
creation of an equation device like the Anchor Test. 


Smith also addresses the problem of the lack of correspondence between tests. 
The High School and Beyond survey is the focus of his comments in which he note* 
problems with the quality of HSB student achievement data and the nature of the 
concepts measured by HSB relative to the methodology used for the testing. 

Smith also notes problems in articulation with International Evaluation of 
Achievement, NAEP, and State assessments. 


Lehnen presents a case study on how one State legislature (Indiana) used NCES 
data to compare resources and performance between that State and other States. 
In the area of output measures, Indiana used three sets of NCES-supplied 
numbers: 


o Median years of education, 
o Percent graduating from high school, and 
o Average SAT scores (for 22 states). 



:redible as national statistics. (This observation supports Hersh's call for 
:ase studies.) 


■urther, Lehnen advocates that NCES determine what kinds of measures should be 
collected via a public process that involves interest groups, policymakers, and 
iducation professionals. "Without such information, the Nation's policymakers 
:annot effectively evaluate the Nation's schools and develop programs to remedy 
leficiencies." 

alberg urges that new tests and testing procedures be developed that take 
advantage of the technology of the moment (computers) and the concept of 
'tailored-testing" which adapts test items to students. Under this approach, 

;he most discriminating test items would be assigned to each student so that 15 
Items would yield scores as reliable as 90 batched items suited to the average 
student. Smith and others believe that the use of computers should enable one 
:o assess the higher order skills that go beyond the basic skills tested by 
1AEP, USB, and IEA. 

?he need for definition and measurement of critical thinking and higher order 
skills is a recurring theme. Buccino questions whether current tests measure 
ligher order skills. Like Smith, Scott-Jones urges the development of 
ppropriate test items. (See also Thomas, Bishop, B. Turnbull.) 

Subanks argues that tests to assess higher order skills do exist. He describes 
:he Degrees of Reading Power (DRP) developed by the College Board and the Word 
Pest being validated by Carver to evaluate reading comprehension as opposed to 
aerely sounding out words, and the Lauton Formal Operations Test to evaluate the 
evelopment of thinking skills as opposed to rote memorization. 

ilkinson calls for NCES to play a strong role in developing classroom learning, 
Jevelopment, and achievement indicators. In addition to group and individually 
idrainistered tests, she calls for tools to assess social and communicative 
ichievement, differences in achievement due to cultural and situational factors 
md "direct observation of students naturally occurring behavior in a variety of 
classroom situations," asserting that "this knowledge mediates both the teaching 
ind learning of academic subject matter in classrooms" by children. 

[n contrast to the majority of writers who favor large scale data collection, 
lersh takes a different approach to the need for outcome information. While 
mpporting collection of data on standardized tests, Hersh argues that the only 
/ay to effectively measure higher order learning skills involving analytical 
Learning is to conduct hundreds of indepth case studies. He argues that the 
result of these case studies will illuminate the meaning of organizational 
sfficacy for a particular school. Hersh argues that the case study approach, 
lsed as an assessment of organizational efficacy, would inform us of what school 
:onditions working together seem to explain student achievement as well as 
student and faculty satisfaction. 


ACHIEVEMENT 

Phere does not appear to be any quick fix to the problems associated with 
iefinition and measurement of achievement. But while solutions are being 
sought, measurement using the available tools must continue. SAT and ACT, NAEP, 


and longitudinal studies such as HSB and ^ the r esu 

information. The users of t ese ^mindf^ ^ & 

really mean and what con us everyone in the education commun 

warning that should be closely heeded by everyone 

-The growth of testing...may be contributing to a narrowing of the...teac 
agenda." 


SAT/ACT Scores. Several writers commented specifically on the um of ^ 
American College Testing scores as an indicator of student ^ 
states that these scores say nothing about the performance of the educati 
system with respect to all students in a State. 


Cronin decries the fact that the SAT scores have been made the "Oow Jones 
indicator of educational achievement." He goes on to say that it is wron 
use these scores in State comparisons because. 


o They do not test commonly taught skills. 


o They yield only verbal and math scores. 

o A different percentage of students take the test in each State. 


o They ignore demographic composition. 


Hilliard states that the use of SAT and ACT scores as a measure of ach^ 
is seriously problematic, but then notes that equally problematic is the 
of a meaningful and viable alternative. He notes that there is need for 
national level measure of performance. 

Bishop notes that SAT/ACT scores reflect racial/ethnic and social backgr< 
differences more than differences in the quality of schools. He warns tl 
thA^purport to measure performance but in fact measure talent and backgj 
could confuse public debate more than enlighten it. 

Murnane joins Plisko in raising problems with the use of SAT/ACT scores i 
in part, that since private school students are included, they they are ] 
indicators of the quality of education in a State. He cites the high pe: 
of prep school students in New Hampshire as an example. 

Bishop and Scott-Jones both note that testing is limited to college-boun' 
students and is unrelated to specific curricula. 

National Assessment of Educational Progress (NAEP) . As might be expecte 
NAEP also received a number of comments, NAEP's tests are more extensiv 
earlier, and occur more frequently than any other. Since the effectiven 
schools is frequently judged according to test scores, the NAEP data are 
extremely important (Peterson). Everyone who refers to NAEP wants it co 
though several writers point out problems or disadvantages. 



Plisko describes the limitations of NAEP's subject-specific approach. In a 
given year, NAEP focuses on a few learning areas so it does not give a 
comprehensive picture about what is happening in the whole school. She alsc 
recognizes that the original design deliberately precluded State- and 
district-level comparisons {for political reasons). Several other respondent 
urge that NAEP not permit State comparisons. Cronin suggests that all State 
included, though this raises problems in cost and timeliness. 

Murnane notes that because of NAEP we know much more about outputs than we c 
20 years ago, but then points out that filling the gap of what we do not knc 
about higher order skills would result in a different view of the national 1 
in student performance. {See also Peterson.) However, the lack of data on 
higher order skills makes it impossible to solve the many puzzles created by 
existing NAEP data. For example, why do the reading skills of 9-year-olds 
increase while those of 17-year-olds decrease? He also poses the problem of 
emphasis on test results influencing curriculum. 

Murnane makes three recommendations about NAEP: 

1. Continue funding as a high priority while retaining current 
plans to increase the frequency of math and science testing. 

2. Supjjort the development of better NAEP tests, including bettez 
multiple choice and open-ended responoo questions. 

3. While it is important to introduce better tests it is also 
important that enough old test items be retained to permit 
comparison of new NAEP results with previous tests. 


W. Turnbull includes an extended quote from Mesoick, Beaton, and Lord, Natic 
Assessment of Educational Progress: A New Design for a New Era . The write* 
stress the need for NAEP to address student competencies, achievement, and 
attitudes, not only to provide a national overview, but also to be relevant 
State and local concerns to assist them in meeting their goals and objective 

NCES Longitudinal Studies . Interest as expressed by the authors is very hie 
the NCES longitudinal surveys. From the comments, it is clear that there is 
great deal of support and excitement about High School and Beyond and the 
NELS-88 surveys. At the same time several reviewers {Plisko, Smith, Buccir 
point out the inadequacies in the data—what is collected and how it is 
obtained, and the limited scope {beginning with 10th grade). 

The council of Chief State School Officers {CCSSO) fully supports both studj 
and finds the resulting data extremely helpful. However CCSSO urges NCES tc 
assure that the State-representative data as well as nationally representatj 
data are produced. 



ir\fc} iiatiyuoi 
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specific. 


they would like to see the data quality improvea a 
o£ content and sample site to make the data more State 


Other comments on these NCES longitudinal studies include the 


following: 


aa 


o More attention should be given to elementary school children. Survey 
should include parents as well (Scott-Jones). 


o Data needs to be collected on extracurricular and work activities of 
students in grades 10-12. 

o NELS-88 should track students from grades 7 or 8 (Smith), 
o NCES should commission a Preschool and Beyond study (W. Turnbull), 
o A broader range of outcome tests is required (Natriello). 


Finally, with respect to the NELS-88 study, which will begin in the near iutu 
Bishop urges that the data sets be large and that NELS-88 be designed to rnergi 
the following kinds of data: 

o Achievement teats on a great variety of subjects, including subtests 
for higher order skills and basic skills. 

o Aptitude tests. 

o School records on courses taken, grades, absenteeiera, special 
services, and test scores. 

o Student background and attitudes. 

o Parent information. 

o Surveys of students, teachers, guidance counselors, and principals. 

o Multiple administrations of tests and surveys at 2- to 4~year 
intervals. 

o Labor market outcome data, including employment and unemployment 
histories up to age 40. 

o Interviews with employers to measure skills and job performance of 
recent school leavers. 


Bishop also urges that the NELS-88 study include a cohort of 2nd-graders and 
their .parents with resurveys in 1992 and 1996 so that they would form all or 
part of the sophraore cohort in 1996. 


achievement provoked responses from a sizeable number of writers. A basic 
problem with international data, and to a lesser degree with much of the 
domestic data, particularly private school data, is that they compare apples 
oranges. Any conclusions drawn from the data must allow for the differences 
the educational systems, populations being served, etc. One writer suggests 
that to balance the comparisons we need to add data to our international 
comparisons that give the proportion of children who attend or are qualified 
attend college {B. Turnbull). 

The Council of Chief State School Officers suggests that data of this sort v, 
be most useful as States analyze their own data sets. 

Although most of those who address this area are quite supportive of 
international efforts, almost all are critical of the long time between 
administration of assessments and obtaining of results. Plisko reports that 
Commission on Excellence, which did much of its work before 1984, had to rel 
international comparison information that was almost 15 years old. Selden <3 
notes that even the most recently released information (post-Excellence rape 
is based on a 1976 data collection. 

Hilliard, in commenting on international comparisons, states a belief that t 
achievement floors in other nations appear to be close to the ceilings in tl 
United States. He recommends that our comparisons should be with industrial 
nations. 

While Walberg notes that the unique organization of the elementary and secon 
system in this country may hinder the collection of statistics, Hilliard raak 
strong case for their collection: 

"International comparisons may be helpful in interpreting 
what we are doing in education and in setting the 
appropriate expectations for what can be accomplished 
in education. ...To the extent that the comparisons 
are valid they force us to raise serious questions 
about our estimates of what the general population 
(of) students in our own nation are capable of 
achieving." 


Both Murnane and Buccino make the point that international comparison data 
viewed at a single point in time are not terribly helpful or reliable. Bott 
argue for examining changes over time to determine the relative position of 
students to those in other countries. As W. Turnbull notes, "Comparisons oj 
student accomplishment in the United States with that elsewhere,..can help \ 
raise our sights in areas where others are doing better and lead us to 
examine...worse results." 

A number of specific recommendations are also contained in the papers: 



o Financial support should be continued with emphasis on test and 
sample designs that permit comparisons over time (Murnane). 

o There should be a regular schedule for IEA test administration 
(Murnane). 

o To improve local cooperation and reduce the nonresponse rate, 
the Council of Chief State School Officers should be involved 
in administering the tests (Murnane). 

o NCES and NIE ehould fund research on international data (Bishop). 


Bishop also makes a strong argument that the international data should be 
more available and suggests that NCES publish a number of additional 
international data tables, including more detail on science and math, rea< 
comprehension, literature, and civil attitudes and education. 

Private and Public School Data . The discussion of private school data is 
intriguing, even though only a few writers address the topic. 

Murnane makes the point that there are in reality three different types o: 
non-public schools; 

o The "traditional" non-public schools, characterized by religious! 
affiliated schools and a variety of other not-for-profit schools. 

o A growing number of for-profit schools. 

o A vast array of "after-hours" schools which often complement publ 
schools. 


Murnane believes that we need data on the number and characteristics of s 
who enroll in this last category of schools in order to better understand 
differences in student achievement scores. 

The National Education Association urges that the data elements in the pr 
and public school surveys be aligned so that the data sets are comparable 
Scott-Jones goes even further suggesting that comparisons on achievement 
between those public school students in college preparatory programs and 
private school population. She believes that if this were done some of t 
higher achievement scores for private school students would be diminished 

Hannaway discusses the need to factor out self-selection effects in measu 
of private schools. She suggests that to distinguish self-selection effe 
from school effects requires a data collection effort that studies a smal 
number of ccoununities over time. 


icott-Jones also raises the provocative point that our data about private 
ichools may be inaccurate. She suggests that the stability of private schools, 
iatterns of movement between public and private schools, and motives for 
ittendiny private schools need to be monitored. 

’Lnally, billiard argues that the small sample of private schools raises a 
[uestion as to whether, considering the diversity of private schools, nationally 
. epresentattve data are available. Hilliard also notes that there lias been a 
ireat deal of attention given to the higher achievement levels of private school 
itudents and yet little data exists on the "types of treatment" offered to 
itudents in each sector, 

temographic Issues . A number of papers deal with what most accurately could be 
lescribed as demographic issues. 

larker presents a paper devoted exclusively to the research and data needs of 
imall and rural schools. He argues that while two-thirds of the Nation's school 
listricto are in rural areas, urban areas received the "lion's share" of 
ittention: "In our nation's guest for excellence in education, the data and 

.nformation needs for small/rural schools must be included." 

lubanks, conversely, addresses himself to the data needs of big city schools, 
le calls for new sources of data to measure the increasingly minority and poor 
>opulutions of the big cities because the traditional measures have proved 
nacourate. After discussing the types of data needed to understand what is 
happening in the big city schools, ho summaries with the point that the data 
mst be usable for improving perforraance. 

Isdan also talks at some length about demographic issues facing large urban 
jchools. He calls for the collection and careful analysis of information to 
.iQploment remedies in our problem-plagued urban districts, 

leveral writers refer to the need for data on Hispanic and other bilingual 
itudents, but Valdtvieso addresses the data needs of the Hispanic population in 
ireat detail. The data noeds of Hispanics are unique for a number of reasons 
starting with linguistic and cultural differences. Valdivioso points out that 
■Hispanics are replacing blacks as the group at the bottom of the education 
.adder in. terras of both attainment and achievement," but statistics are skewed 
;i.nce Hispanic drop-out rates are extremely high with only relatively high 
ichievers remaining in school and thus getting measured. 

Jhon dealing with different segments of the student population, Scott-Jones 
otes that while poor, minority, and female students must be carefully assessed 
:est scores should not be the basis for achievement comparisons. She urges NCE? 
;o include in its data collection efforts assessments that include 
'comprehensive measures of learning and thinking that are appropriate for 
Jinority students." She also urges that NCES monitor differences in math and 
science scores by sex. Finally, she points out that since the numbers from 
linority groups may be small, it may be necessary to oversaraple them. 


The question of data by race/ethnicity and sex is raised by several others as 
well who urge that data be tabulated in a manner that would permit comparisons 
of, Cor example, black females with white and Hispanic females at the sophmore 
level of high school {Scott-Jones, Hilliard). 


RETENTION AND DROPOUT RATES 

An area that provokes some major concern among many writers deals with school 
retention. Criticism is leveled at the two commonly available sets of data, om 
collected by Census, the other by NCES. These two sets of data yield wide 
discrepancies. There is also criticLsm {Scott-Jones and others) that by 
starting longitudinal surveys in the 10th grade much of the dropout problem is 
missed since a high percentage of dropouts occurs before then. Eubanks believes 
that alienation from school begins as early as the 5th or 6th grade. 

As noted earlier in the discussion of longitudinal studies, it is suggested tha 
the NELS-08 survey include a cohort of 2nd-graders who would become the 
sophomore cohort of 1996. Some meaningful data about when dropping out occurs 
should result. 

There are major discrepancies between estimates of the national dropout rate, 
with NCES reporting it at 27 percent and the Census Bureau at 16 percent. 
Figures at the local level appear to be even worse (Plisko). Scott-Jones 
attributes at least part of the problem to definition. Census, for example, 
asks for self-reporting and includes passage of high school ecjuivalency exams e 
completion of high school. NCES measures the difference between the number of 
9th-grade public school students and the number of high school graduates 4 year 
later, which does not allow for dropouts prior to 9th grade or for graduates 5 
or more years after 9th grade, Cronin suggests that a task force of federal an 
State educators should propose a consensus position for defining and thus 
measuring dropout rates. It is important to note that retention rates, i.e., 
the number of students that stay in school and graduate on time, cannot be used 
to extrapolate dropout rates. Several factors, such as early dropping out, 
moving, and late graduates, make this impossible. 

Scott-Jones cites the need for information about teenage pregnancy as a factor 
in the reason females leave high school. Banner urges that dropout rates need 
to distinguish between the various reasons for dropping out, e.g. "stopping 
out," moving, etc. Thomas urges that dropout data include information on 
race/ethnicity, sex, social class, academic achievement, attitudes toward 
school, reasons for dropping out, and expected resumption of schooling. 

Valdivieso talks about the need for data on Hispanic dropouts that consider the 
fact that, because of language difficulties, almost 25 percent of all Hispanics 
enter high school over-age. Many of them reach age 16 before they get to 10th 
grade. 

Harrison is one of several writers who calls for separate statistics about c»ED 
recipients. 

The National Governors' Association (NGA) points out the problem of evaluating 
the post-high school experience of dropouts since they are no longer traced 


in tne several uiacussions aoout aropouxs, tno empnasis is on gatnenng 
meaningful data about the numbers. The second issue seems to be when studer 
drop out. A careful reading reveals an additional concern—why. The 
longitudinal studies, if they begin early enough, should address this import 
question. {See Grant, Thomas, Plisko, Scott-Jones, and Smith). 


NQNCQGNIIIVE OUTCOMES 

Thomas, Bishop, Scott-Jones, McPartland, and the National School Boards 
Association {NSBA) all express concerns about obtaining data on what is best 
described as noncoynitive outcomes. 

Scott-Jones notes that increasingly, schools are relating athletic participa 
to academic achievement. NSBA suggests that we need to measure such ideas c 
entrepreneurship, patriotism, and racial tolerance. He believes that this c 
the public better information than the resource data which seems to predomir 

Thomas urges that NCES collect data on student attitudes toward school on a 
longitudinal basis. Bishop believes it important that we know how well our 
schools do in developing work habits and self discipline. Scott-Jones and 
Bishop urge collection of data on drug and alcohol usage. McPartland talks 
about the need to measure coping skills. 

Buccino stands virtually alone in addressing the issue of citizenship direct 
He calls for "a program for monitoring the adequacy of the education system 
producny leaders, technical specialists, informed citizens.” B. Turnbull 
briefly mentions that data concerning the voting records of recent graduates 
would be interesting. 

McPartland discusses the need for expanding the opportunities for students 1 
provide services to their communities. Cronin includes citizenship in his ] 
of 12 subject skills that policymakers need information about. When discuss 
the need to measure other areas of educational productivity, he refers to t) 
proposal of the Carnegie Foundation for the Advancement of Teaching to requJ 
120 hours of community service. The National Governors' Association lists 
community behavior that affects student outcome as one of the areas they wai 
information about. 


IMPACTS ON NONEDUCATIONAL SECTORS AND LIFE OUTCOMES 

The outcomes discussed to this point in this chapter are the immediate resu; 
of schooling—the measures of what was learned. The impacts on noneducatioi 
sectors and life outcomes are more far reaching and may involve many factor* 
beyond what is learned in the classroom. A brief discussion of these impact 
follows. The next chapter includes more detailed analysis. 

Economic Impacts . One of the most consistent debates in our society for tin 
last two decades has been to what extent does education impact future 
income--and how. 


an education system is to P re P information that relates schooling 

them earn a living, we nee mu NELS _ 8 8 be used to continue the data 
careers and income. He urges that NELS wo oe u* 

collection In NLS-72 and ..SB on Income end urges that the MLLS 88 trace 
individuals who drop out or transter to a ditferen 


Scott-Jones raises the issue of the education that poor and minority chi. 
receive to prepare them for work, and then elaborates on ^a nature of 
employment. She notes that our service society in the next decades will 
generate large numbers of low-payiug, potentially unsatisfying jobs. Sh, 
therefore, urges that data be collected on the nature of employment, as 
on the rate of employment. 


Smith urges NCES to attend to the equality of our educational system and 
relationship to the nation's productivity and national defense. 


The National Governors* Association points out that "as more states move 
preparing a better educated workforce to encourage economic development, 
issue of identifying student outcomes emerges as more than assessing stu 
achievement. More data than test scores, such as the SAT, are needed to 
determine post-school experiences.*' 


Natriello takes a quite different tack. While concerned about preparati 
employment, he urges NCES to broaden its data collection effort to inclu 
sample of employers. They would be asked to provide data on how recent 
graduates fared on performance tests administered by employers, on attit 
toward work, and on what he terms "deportment." 


Readiness for College . Very little is said about the issue of readiness 
college per se, though a great deal of the comment about other issues 
{particularly the SAT/ACT scores) relates to this issue. 

The repeated complaints about the narrow scope of the longitudinal studi 
NELS-88) also talk to the problem of tracking students as they progress 
high school to the work force and/or higher education. Usdan specifics! 
for efforts to bridge the gap in information about secondary and higher 
education and their overlap. Thomas points out the need for additional 
better data about the transition rates from high school to college. 


Others talk about the need for data on the types and numbers of courses 
take and their usefulness for college, but these points must be extrapo! 
from discussions about curriculum issues {Usdan, Thomas). 


Impact on Life Chances . As with readiness for college, very little is t 
directly about education's impact on life chances. A great deal, howev< 
be inferred from other discussions. The frequently voiced concerns aboi 
dropouts cannot be separated from the effect of dropping out on life ch< 
The writers who delve into the demographic issues mentioned above also 1 


this abstract concept which is, after all, rather difficult to measure 
quantitatively. Similarly, several of the respondents who decry the narrown 
of the longitudinal studies seem to be saying that they want to go beyond hi< 
school and even college to see what happens next. 

Bishop does address the question of impact on life chances directly. He sta 
that the "personal efficacy scale measuring the belief that one can control 
one's future is probably the best documented and most researched of the 
sociopsychological scales contained in HSB." He goes on to explain the posi 
correlation between self-efficacy and labor market success. 


SUMMARY 

It is apparent that data about Educational Outcomes are of vital importance 
everyone involved in the educational process. Despite inherent flaws in the 
current systems for measuring outcomes, they must be continued while solutio: 
to the flaws are sought. Particular attention needs to be paid to the accur, 
comparability, and timeliness of the data. Caution must be exercised to avo. 
inappropriate uses and interpretations of data. We must be constantly aware 
the correlation between learning and measuring learning—are we measuring wh. 
we learn or are we teaching what we measure? 

While virtually everyone addresses outcome issues at least peripherally, a f 
respondents turned their attention to this crucial area almost exclusively, 
papers submitted by Bishop; Buccino; Cronin; Murnane; Plisko, Ginsberg, and 
Chaikind; Scott-Jones; Smith? and Walberg should be read carefully before an 
conclusions are drawn concerning collecting data about and measuring educati 
outcomes. 


IMPROVING OUR UNDERSTANDING OF THE RELATIONSHIP BETWEEN 
EDUCATIONAL INPUTS AND PROCESSES, AND EDUCATIONAL 
OUTCOMES AND LIFE CHANCES 

Margaret K. Gwaltney* 

Betty Balcorab** 


Lous chapters of this paper have noted a variety of educational data needs 
sated by the writers of papers invited by NCES. Indeed, they range from 
: data to process data to outcome data, or from information about financial 
irees and numbers of students to teacher characteristics and curriculum 
mt to data on achievement, attainment, and life outcomes. However, several 
3rs {Berryman, Buccino, Bishop, Hawley, and others) carry the data 
action need one step further. They argue that the relationship between 
itional inputs and processes and educational outcomes needs to be better 
cstood. Their position is that without this understanding, it will be 
amely difficult to improve education and school effectiveness in forthcoming 

3 . 


writers go even further in their argument. The ultimate goal of education, 
no states in his paper, is to produce leaders, technical specialists, and 
:med citizens and to address long-held equity concerns. Thus, while 
ational outcomes such as achievement and retention and their relationship to 
ts and processes of education are important, the impact of schooling and 
rnative forms of education on the lives of those who go through the 
Jtional system is perhaps of even greater concern. 


jaret K. Gwaltney is a Senior Program Analyst at University Research 
oration where she works on projects in the fields of education and work 
a productivity and effectiveness. Prior to joining URC, she worked at Abt 
tiates and the Rand Corporation in their Washington, D.C. offices. 

tty Balcomb is a research associate with University Research Corporation, 
s a graduate of Adelphi University and has done graduate work at Columbia 
arsity. 


The implication, of these and the Census Bureau 

federal agencies, such as the Department of Laoor an SDecific < 

many, even though greater individu, 

elements are needed. We must find out about the “ ln educati , 

their education, what happens to individuals mo p / , econom 

vhat the impact of education is on general social, cultural, and econom. 

affairs (Buccino). 


A number of the writers (Buccino, Hawley, McPartland, Hersh and others 
models of schooling and educational excellence and suggest those as fra 
for data collection and research. One of these models (McPartland) is 
Figure 1. As McPartland states in his paper, the model may be viewed a 
"comprehensive account of instructional and organizational choices made 
schools and school districts that are likely to have important consoque 
student academic achievement, personal growth, and school related atttt 
social behavior." This model and others are a starting point from whic 
education community can begin to collect data and conduct research to u 
the relationships between inputs, processes, and outcomes. 


This chapter summarizes the recommendations made by writers on the topi 
relating educational inputs and processes to educational and life outco 
will become apparent when reading this chapter, far fewer writers deal 
issue than with some of the other issues presented in this synthesis, 
because of the saliency of the issue to educational excellence, as disc 
many of the writers, and the important points made on the topic, an ent 
chapter is devoted to a summary of such recommendations. 


THE DATA AND MEASUREMENT ISSUE 


One purpose of educational statistics is to "facilitate the improvement 
educational policies and practices" (Hawley). More generally, however, 
purpose of these statistics is to provide an understanding of which edi 
policies and practices have a positive effect on educational life outcc 
then use this information to affect improvements. 

The utility of educational statistics for this purpose, however, depenc 
entirely on an understanding of the linkages between certain education* 
and processes and educational and life outcomes. To make improvements 
schooling, we must know what factors to measure and then collect data i 
in one way or another related to educational excellence. 


A Model of the Dimensions of School Instruction 



• Educational climate 






The chicken and egg problem is immediately apparent. Because we do no 
ine cnicxen emu « yy y „ t* difficult to design a data 

Cull understanding of these linkages, it . t , al nractitione 

collection program that helps policymakers and educational practitione 

sews Moreover, as Hawley states, "statistical gathering and analy 
is uninformed by ^search encourages simplistic conclusions about the 
student performance." Many of the writers (Grant, Hawley, McPartlan , 
others) argue that there is a very real need for research that begins 
the questions of educational quality and excellence and questions abou 
effects of schooling on the labor force. This research should not onl 
what are the appropriate data, but also on how these data can be valid 
measured and specified. 


The recommendation on the part of several writers to conduct research 
determine the causal relationships between inputs, processes, and outc 
not diminish the need, as well, for descriptive information. As McPar 
writes, "although the most important questions on this topic are causa 
nature—how do different organizational and instructional practices af 
achievements, attitudes, and behaviors of different groups of students 
lack basic descriptive data about the factors discussed in the model, 
important contribution to NCES would be to provide descriptive informs 
part of an effort to understand how these factors affect schooling out 
Therefore, just because some of the descriptive data may ultimately be 
have little impact on educational and life outcomes does not mean that 
elements described in previous chapters of this report are not valuab] 
not contribute to a better understanding of schools. 


QUESTIONS 01T PRIORITY 

The general interest in improving educational quality seems to imply c 
standard--a common goal--toward which educational practitioners and pc 
are aiming. No standard exists, however. Individuals—policymakers, 
practitioners, and the general public—each have their own conception* 
the education system should be doing for children. Each has establisl 
of priorities for the education system. Scott-Jones even questions w) 
set of educational outcomes is appropriate for all children. 

Hawley says that data collection for school improvement should be guic 
questions: 


1. What are the problems or issues we want to understand better; and 

2. what theory or theories might best identify the range of factors 
that influence the outcomes in which we are interested? 

lawley suggests answers to both questions. The answer to the first, he says 
ihould be to ’’know how to most cost-effectively improve schools." The answe 
;he second question, he admits, depends on how one determines whether a scho 
las improved. 

? or Hawley, the "central purpose of schools is to produce student learning." 
Phen "improvement would be measured in terms of amounts or rates of learning 
it least, in terras of the relative achievement of students once all of the 
variables schools cannot influence are taken into account." But he acknowled 
:hat the answers to the above questions may be different for others and "bee 
scholars disagree about the major determinants of school effectiveness, and 
because the importance of different factors depends on the outcomes of schoo 
/ith which one is concerned, the model or models which would guide federal d 
sollection and coordination efforts should be derived from a consensus of 
Leading researchers and practitioners." 

Certainly, this is one way to begin the data collection effort in the absenc 
research that provides greater information into the causal relationships, 
lawley suggests that exports be asked to comment on alternative assumptions 
about variables and their interrelationships and that this process of specif 
he data collection effort be "interactive until agreement is reached either 
about particular variables or alternative explanations Cor specified outcome 

3o far in this chapter, we have focused on educational outcomes. Yet, if we 
turn to a discussion of lifetime outcomes, we see that the concern about 
Priority is again relevant. Although policymakers and practitioners agree t 
</e want to understand "how differences in content and levels of investment 1 
to differences in outcomes beyond educational performance and educational 
attainment" (Hawley), the most important of these other types of outcomes 
(whether they are occupational success, income, or something else) is debata 
Different individuals have different priorities and, depending on their 
priorities and the causal relationships that are found to exist between thes 
outcomes and educational inputs and processes, will support different 
organizational structures, changes, or curricular emphases within schools. 

ANALYSES OP EFFECTIVE SCHOOLS 

within the last several years, we have seen the attention of policymakers an 
practitioners turn to the problems facing American education. There have be 
eports on student achievement in the basic skills, published statistics on 
rates of functional illiteracy, figures on average teacher salaries, compari 
cf the time spent in school by t/.S. students with the time spent by their 
counterparts in other countries, and so on (Selden). The ultimate concern c 
3ach of these inquiries is what conditions, what factors, make schools 
effective—what can be done to improve the quality of American education. 


The .mentions being asked are difficult ones, and the answers are not 
apparent. Research must be conducted on effective schools, if we hop* 
influence today's schools and make them better at educating our child: 
must determine what the important factors are m making up an effects 
Then, and only then, can the variables be defined, and the data colle< 
is only after such data are available that we will truly be able to cl 
schools for the better and be able to answer the further question of 
create the necessary conditions for school improvement. 

What we do in the meantime is key. Vie cannot just wait until the res< 
conducted, the variables defined, and the data collected and made ava: 
must begin collecting data on outcomes and relating these outcomes to 
and input data. With the research findings currently available, a nui 
writers suggest models that may guide NCIiS and other federal ayencios 
collecting data that can assist policymakers at all governmental leve 
the conditions of education. Hersh, for example, breaks the attribut. 
effective schools into two groups in his model: (1) social organizat 
defined as those characteristics that pervade the school building, an< 
instruction and curriculum. Attributes of the social organization th< 
pertain to effective schools include (Hersh): 

o Clear academic and social behavior goals 
o Order and discipline 
o High expectations 
o Teacher efficacy 
o Pervasive caring 
o Rewards and incentives 
o Administrative leadership 
o Community support. 


Hersh labels these attributes as necessary social conditions that helj 
and students to excel. 

Important attributes under the heading of instruction and curriculum, 
model, are the following: 

o High academic learning time 
o Frequent and monitored homework 
o Frequent monitoring of student progress 
o Tightly coupled curriculum 
o Variety of teaching strategies 
o Opportunities for student responsibility. 


It is the cumulative effects of the above conditions that is likely t 
highest payoff, in terras of school effectiveness, not merely the deve 
one or two of these characteristics at random. Hersh labels the cumu 
of conditions necessary to promote student achievement "organizationa 
efficacy." This theory of organizational efficacy, he argues, needs 
subject of greater study. A more qualitative rather than quantitativ 
assessment of schooling is called for. 


McPartland supports Hersh's line of thinking. He also suggests a model of 
school factors and student outcomes that relates student learning to school 
organization and policy and instructional practices. Classroom conditions h 
the ''most immediate impact on student learning and development," but these 
oonditione are "facilitated by school organization and policy" (McPartland). 
argues that certain combinations of school organization and policy will 
encourage school effectiveness, while other combinations will not. The 
relationship between school organization and classroom conditions is not a 
causal one, but rather a facilitating one; it is an "enabling or supporting 
connection rather than a determining connection" (McPartland), 

Despite the number of theories that exist about what makes an effective scho 
ittle research is currently available to make any definitive statements. T 
•/hile the writers recommend collection of data about school organization and 
structure and classroom practices, they offer no firm evidence that one or 
several of the variables associated with these general headings are the 
determining factors in educational effectiveness and excellence. Data 
collection must proceed along with research into the question of school 
sffectiveness. Upon the completion of each research study, variables that h 
significant impact on school effectiveness may become apparent, and more 
relevant data may be collected and used as a basis for making informed polic 
decisions about improving the quality of education in the United States. 

'JOES and other federal agencies tasked with educational data collection Ghou 
also provide policymakers with indicators of the quality of education 
(Scott-Jones; Selden). These indicators would allow policymakers to compare 
educational quality among States and between the United States and foreign 
countries. The indicators should reflect not only achievement in basic skil 
out also in higher order skills and even such subjects as music, art, and 
anguages other than English (Scott-Jones). Subjects such as physical and 
utritional education might also be considered in the assessment ot' student 
achievement. 


ANALYSFJS OF EFFECTIVE TEACHING 

Effective teaching is one factor that makes effective schools (McPartland; 
uccino; Hersh; and others). It should be considered in any investigation o 
educational excellence and effectiveness. Yet it is also a legitimate conce 
n and of itself. The quality of the resources within the educational syste 
extremely important. In fact, as Buccino notes, the preprint of Indicators 
Educational Status and Trends emphasizes the importance of the quality of th 
teaching force. And, although there have been many reports of the inadequat 
supply of teachers, particularly in the fields of mathematics and science, i 
clear from the data that there are not too few people with teacher certifica 
n those fields, but rather too many with certification who choose to go int 
fields other than teaching. Investigations and research must therefore exam 
the question of why the teaching profession is not as attractive to these 
ndividuals as other careers. Particular attention needs to be paid to teac 
salaries. 


force. Many of the current educational reforms affect or will affect 
teaching force. Legislation now being considered in several states m, 
the requirements for teachers and ultimately the flow of teachers int< 
of teaching (Berryman). It will "affect the stock of teachers—their 
field of teaching expertise, and quality." The reforms include salary 
changes in high school graduation requirements, and competency tests ; 
teachers. 

Reforms are currently being considered without much information about 
teaching force. Berryman likens this to flying blind: "When this la] 
is not conceived of as a system, when the data do not exist to diagnoi 
with this system at appropriate policymaking levels (national, State, 
district), and when the data do not exist to monitor, let alone projei 
particular reforms affect the behaviors of potential, new, and experi' 
teachers, we are flying blind. Under these conditions the chances of 
misdiagnosing problems are high; the chances that reforms will produc 
changes that reformers want are low." 

To carry out reforms wisely, the following data, at a minimum, are ca 
The quantity, or number, of teachers 
The quelity of teachers 

The number and quality by level; a level being defined as a g 
category that requires at least some teaching skills not requ 
other categories 

The number, quality, and level by field; field being defined 
teacher's special field of training, such as mathematics, bil 
education, biology, English, or art 

The number, quality, level, and field of teachers by their 
race/ethnicity 

The number, quality, level, field, and race/ethnicity of teac 
their geographic distribution, the geographic units of intere 
districts, states, and the nation and unique supply and deman 
environments, such as inner city schools or rural schools. 


These data can be collected from one or several of the following sour 
(1) schools and boards of education, (2) public administrative and le 
groups at the State level, such as State education agencies, State be 
education and education committees in State legislatures, (3) associa 
as teacher organizations, associations of teachers of particular subj 
State School Officers, and associations that represent schools in par 
environments, such as the Council for Great City Schools, and (4) dat 
collection agencies or organizations, such as NCES, the Bureau of Lafc 
Statistics, the Bureau of the Census, Educational Testing Service, th 
Center for Education Information, and so on (Berryman). 


Jconoinic outcomes are defined as Labor market status, occupation and income 
Bishop; Plisko; Walberg; NCRVlS). They are important outcomes of the school 
>rocess. However, little research has been conducted on the relationship 
)etween schooling and economic outcomes. Although Bishop notes that the Dlg< 
>f Education Statistics provides some data on economic outcomes, it is quite 
.imited. The Condition of Education provides no data on economic outcomes, 
leither of these reports shows data on the economic outcomes of those who ha' 
completed one or two-year occupational programs or who hold associate degree: 
Bishop). And, even the few data that are reported fail to show the 
•elationship among educational inputs and outputs and economic outcomes. 

)ospite the limited reporting of these data, some research has been conductec 
;his line of inquiry. One study (Bishop, 1985) shows that achievement test 
jcores have little impact on employment and income. Another (Olneck and Bii: 
982) shows that improved performance on standardized tests "accounts for on; 
small part of schooling's total impact on labor market success" (Bishop). 

’he findings from these few studies are provocative. If one objective of 
jchooling is to produce individuals who can and will succeed in the labor 
market, should the emphasis be ott basic skills and on preparing students to 
score well on achievement tests? When one looks at the goals of vocational 
education—e.g., job readiness, training in occupational skills, good work 
ittitudes--there is reason to be particularly concerned that the emphasis on 
icademic subjects to achieve labor market success may be misplaced. Bishop « 
>thers argue that "data on the economic outcomes of elementary and secondary 
sducation are essential." 

4CES or other national organizations must take the lead responsibility for 
collecting economic outcome data and investigating the relationships between 
:hese outcomes and educational inputs, processes, and outcomes. Bishop 
recommends that NCGS get support from the American Association of Community . 
runior Colleges, the Middle Atlantic Career Counseling Association, and othe: 
md contract with the College Placement Council or the National Center for 
lesearch in Vocational Education to conduct a salary offer survey in two-yea 
institutione (Bishop). 

?ime series data are also needed (Bishop)--data that show the labor market 
success of those who have left school over time. These data would help 
policymakers and practitioners assess whether declines in SAT scores and oth 
Indicators of academic achievement have resulted in a parallel decline in 
luality of jobs obtained and unemployment rates. The following data, in 
particular, are needed according to Bishop: 

1. Unemployment rates and employment to population ratios of young 

people who graduated from or dropped out of high school during the 
previous year and are not enrolled in school (see Table C43 of Labo 
Force statistics derived from the CPS: A Data Bank, 1982), If 
possible, separate scores for blacks and for single females should 
published. The table should also contain an unemployment rate for 


2. Indicators of the quality of jobs obtained by recent high sch 
graduates and dropouts. The following are recommended: 

Industrial Composition 

-- share of jobs in manufacturing, mining, constructic 
transportation, and public utilities 
share of jobs in government 

share of jobs in wholesale, retail, or service 

Occupational Composition 

share of jobs in laborer or service occupations 
3hare of jobs in operative occupations 
— share of jobs in clerical occupations 

-- share of jobs In sales occupations 

These data should be presented both for those who graduated a 
who dropped out of high school the previous year, and for all 
year old high school graduates not enrolled In college and fc 
18-24 year old high school drop outs. 

3. Average yearly earnings of young people not enrolled in schoc 

categorized by educational attainment. Three age groups shou 
reported: 18-24, 25-34, and 35-65. The educational attainme 

categories might be 0-11, 12, 13-15, 16 and 17 + . Separate da 
should be provided by sex and for full-time, full-year worker 
order to accentuate the comparisons across educational levels 
information could be presented as ratios to the earnings of Y 
school graduates (real dollar amounts of earnings would be pt 
only for high school graduates). Tabulating these data for i 
school leavers and comparing it to average weekly earnings of 
groups will provide a measure of the relative labor market si 
those who have recently completed their schooling. 

4. Unemployment rates for people categorized by education and by 
(e.g. , 18-24 and 25-65). 

These time series data will also help to distinguish long-term trends 
return to vocational and other types of education from short-term shit 
to the business cycle (Bishop). 

Data are also needed by States. Theso data should compare "labor mar) 
success of groups with differing amounts or kinds of education, but si 
tho same economic climate." Particular need exists for data that shov 
increases in educational achievement measured by graduation from high 
completing some college, and completing a four-year degree, improves c 
labor market success in that State. (Admittedly, this is an imperfect 



Earnings of different age groups (18-19, 20-2 1, 22-24, 25-29, 30-34 
35-4 4, 45-54, 55-64, 65+) by years of schooling completed by sex, In 
Minority status and for all workers and full-time, full-year worker/ 
(Available in Table 237 of Detailed Population Characteristics of 
the 1980 Census.) The focus of the table should be the earning ratit 
for people with differing educational attainment. 

Occupa tional and industrial distribution of people categorized by 
educational attainment, age, sex, and minority status. 

Unemployment rates and employment to population ratios by people 
categorized by educational attainment, age, sex, and minority status 


Bishop further suggests that several questions bo added to the 1990 Census 1 
aid such comparisons: questions on the field of study in high school and 
college, degrees received, State in which the individual attended high schoc 
and State in which the individual attended college. 

Other writers argue for collection of data on outcomes that go beyond the 
somewhat narrowly defined economic outcomes just described. Buccino would ; 
more information to be available on such concepts as self-worth, participat: 
in community affairs, and life styles, as well as the impact of education oi 
general social, cultural, and economic affairs. Hawley would like more 
information on the relationship between educational processes and outcomes i 
such lifetime outcomes as participation in the political and social life of 
communities, incidence of antisocial behavior, family stability, and the 
condition of physical and mental health. 


CONCERN FOR EQUITY 

Long a concern of educational policymakers and practitioners has been the 
promotion of equity within the schools. To achieve equal educational 
opportunity, however, requires that policymakers and educators receive data 
about areas of "inequity or potential inequity in schools" (Hilliard). Ho i 
that "traditional areas where inequities appear to occur in school settings 
include such things as differential dropout rates among groups of students; 
transiency rates among teachers and students; differentials in the distribu 
of teachers in assignments by teacher preparation and experience; different 
in expenditures per child" and so on (Hilliard). Moreover, he argues for 
greater examination of the differences in achievement ecores for different 
groups of children, to assess whether these differences are a result of 
diversity in the treatment of children. 

While the writers state that no data collection effort is likely to remedy 
situations where inequities in education exist, much less inequities within 
general society, they do say that it may be "possible to spot situations th 
call for closer examination. For example, if it is shown that teachers who 


Hilliard recommends that the following types of data be collected: 

o more complete data from private schools and the collection of 
same types of data from private schools as from public schools 

o data on access to data processing equipment for computing, wor 
processing, and instructional software; data on amount and typ 
paid or unpaid after school tutorial or enrichment services 

o indices of mobility for teachers, students, and line site 

administrators; data on mobility regularly as part of the cens 
sampling effort at least for the elementary school years, data 
amount and type of pre-school experience to which students hav 
exposed 

o data on the mobility of students in and out of special educati 
category of service, over time 

o data on the performance of students on international tests of 
achievement 

o data to permit race by sex analyses 

o disaggregate data, to enable reporting on both aggregated as w 
disaggregated data 

o data on the academic major and minor preparation of certified 
disaggregated to the school site level 


ALTERNATIVE LEARNING OPPORTUNITIES 

Most of the writers focus their recommendations on some aspect of trad 
schoolina- However, there are several (Cronin, McPartland, Natriello, 

alternative learning opportunities and the need t 
"acting data about these activities. These write 
:he acquisition of skills occur in a variety of 
tool. They want some meaningful measuree of the 
e opportunities have on students and young adul 
>uld he to assess more clearly the separate effe 
es of learning opportunities. 

express an interest in finding out about partic 
ies, athletics, and activities such as scouting 
itional school arena. McPartland, for example, 
urricular activities available within most high 
otivities be extended to the middle school, as 
Jtion of a wide range of extracurricular, cocur 


effects ot tnese activities on student development. 


Cronin calls for measurement of extra- or cocurricular activities and the r< 
of participation. He states that authorities have agreed for a long time tl 
students learn from activities such as serving on the student council, 
participating in the debate club, band or chorus, playing basketball and otl 
sports, and participating in other school activities. 

Scott-Jones writes in Her paper about participation in athletic activities « 
their impact on academic performance. She argues the need to study further 
relationship between participation in athletics and academic performance, it 
part to substantiate or give reason to advocate maintaining minimum academic 
standards for participation in athletic and other extracurricular activities 

other writers talk about activities outside the school setting. Buccino ta! 
about measuring the effects of informal learning—e.g., casual reading, 
television, museums, libraries, and involvement with community organizations 
He writes, "information regarding education in out-of-school settings shoulc 
studied. He have already mentioned television and museums as focal points, 
libraries and community groups such as Boy Scouts and Girl Scouts should all 
taken into account." 

Miller devotes most of her paper to the question, "Where is the library/med: 
center in alt of this...." She points out that we need to know what learning 
going on in or being facilitated by libraries more than we need a count of ' 
materials on the shelves. 

Bishop calls for data on the labor market behavior of students. He recoramet 
that information on the effects of amount of time spent working on educatioi 
outcomes be studied. The question of whether the financial benefits are wo) 
the reduction in time spent studying, he says, needs to be answered. 

Berryman feels that discovering what learning opportunities—whether tradit; 
or alternative—are interchangeable in producing skills is one of the keys 
answering a number of policy questions. She would like the longitudinal sti 
to support research in this area. Berryman cites a study on employment in I 
(electronic data processing) occupations which shows that formal education « 
substitute for work experience or vocational training, and concludes that 
"trainable" individuals can enter higher skill occupations freely. Her ult: 
conclusion is that "substitution possibilities pervade the educational syst< 
and the work place." 

Natriello begins his list of recommendations for the redesign of NCES ’ s dat< 
collection program with the following: "NCES should explicitly consider mo 1 
beyond the collection of data on schooling to the collection of data on 
education." Although he argues that schooling should remain at the core of 
agency’s data collection efforts, more emphasis than currently exists shouli 
placed on "educational activities that extend beyond formal schools." The 
educational "phenomena" mentioned by Natriello as legitimate concerns for a 
collection program include the mass media, educational software and other n 
information technologies, and supplementary instruction from proprietary sc 
and tutoring services. 


Natriello further suggests that the effects ot tnese external ana axternatl 
sources of education may serve as important control variables in evaluating 
school effectiveness, similar to parent educational levels and economic 
resources. The following items, he suggests, might be included on NCES sur 
to parents and students: 

o private lessons in music and/or art 

0 private instruction in sports and other physical activities 
(e.g., tennis, horseback riding) 
o participation in a computer users group 
o training related to a part-time job 
o attendance at an ACT test preparation course 

o remedial or supplementary instruction in one or more school courses 
o training provided by a youth or community group, such as the YMCA o 
the Boy Scouts or Girl Scouts 

Data on participation in these non-school activities would help determine t 
impact of participation on student performance and academic achievement. 

Two other strategies for collecting this type of information are suggested ' 
Natriello. These are collecting information from educational services to t 
community and collecting information from a representative sample of commun 
and the non-school educational programs in those communities. This strateg 1 
says "would permit analyses to determine the distribution of supplementary 
educational activities across communities with different demographic and 
economic characteristics.” The point of this sort of data collection is to 
"begin to understand the extent to which non-school activities contribute t' 
development of elementary and secondary etudente in the United States.” 

Finally, Murnane notes that more and more children are attending private scl 
after their day at the public school has concluded. The instruction they 
receive at the private schools is intended to supplement their instruction 
the public schools. Murnane recommends that NCES learn more about after-hoi 
private schools in the United States. He suggests including a set of guest 
on a new longitudinal study of American students that ask whether students 
attend after-hours schools and, if they do, what the schools do and what th- 
cost. He recommends that the Census include a similar eet of questions on 
October CPS survay. 


RECOMMENDATIONS REGARDING ANALYSIS AND INTERPRETATION 

Nearly three decades ago, the United States received a serious blow to its 
self esteem when the Russians launched Sputnik, becoming the first nation 
aunc an artificial satellite. The reaction included sharp focus on our 
sc ools particularly the weaknesses in science and math. A period of ref 
followed, but before long we slipped back into complacency. 

ft . rc Riek , the report of the Commission on Excellence in Education, 

n a blow to the Nation. We are in another period of reform 

exam ne what is happening in the schools, what effects the various 


sdan cites Kirst's issue attention cycle in his paper as he alerts readers t 
he problem: (1) alarmed discovery, (2) crisis activity, (3) disillusionment 
ith results, and (4) return to neglect. He then warns that the current refo 
ovement is rapidly passing tlurough phase 2, and says that the movement can o 
ndure if the effectiveness of specific reforms can be proved. 

ducat ion a 1 policymakers at all levels, as well as the general public, theref 
eed to know how well the educational system is working (Cronin, Plisko, 
ishop). They can no longer make do with simple, short-term, quantitative 
easures of inputs and outcomes. They need long-term measures of progress an 
nterpretations of changes. As Hawley states, we need data that focus not on 
he condition of education, but on explanations for that condition. William 
urnbull further warns that we must avoid swamping audiences with data, the c3 
ust be synthesized and interpreted. 

ata analysis and interpretation, however, are not easy tasks. Aside from th 
eing a time lag between data collection, analysis, and then reporting, NCES 
n organization has ventured very little into this type of activity. Moreove 
he same problems associated with the simple collection of data are applicabl 
ven more to analysis and interpretation: questions of definition, 
omparability, accuracy, and timeliness. In addition, there is concern about 
lie burden placed on the suppliers of information. 

ow far NCES should go in providing analysis and interpretation is also 
ertainly a legitimate question and one that is acknowledged by the writers ( 
urnbull). Jluccino urges NCES to engage other programs in the Department of 
ducation and other agencies to interpret data from NCES. He then goes on to 
all for a series of substantial interpretive papers to be published along wi 
ata sets. 

espite many stated and implied reservations, the writers voice considerable 
emand for interpretive information. The discussion that follows breaks the 
eeds expressed in the papers into three distinct categories of analysis: 
ost-benofit studios, program evaluation, and State comparisons. 

ost-Qenefit Studies . Several writers advocate cost-benefit studies to (t) 
valuate the impacts of various reform programs, and (2) measure the earning 
apacity of students after graduation. The call for cost-benefit studies is 
new one. Whenever public funds are used to support an activity (in this ca 
ducation), justification for spending these monies is demanded. The writers 
ontend that the public is willing to fund education "only if there is more 
ducation for the dollars" (Usdan). To earn continued support from the 
axpapers, schools will have to justify what they do from a cost benefit 
tandpoint. Moreover, the public will continue to pick up the bill for 
ducation only if the educational reforms that are being implemented today ar 
hown to be cost-effective. 


t'j.j.aN.o ecnoes usuairs point, sayxuy timt we neeu tu lueoauLB wuuuiui we ai« 
getting our money's worth from educational reforms. Hawley, too, points out 
that comprehensive data collection efforts like High School and Beyond will i 
more valuable if the data allow analysts to understand the economic costs of 
improvement strategies. 

Each writer who speaks to the subject of cost-benefit studies presents his 
position somewhat differently, even if the ultimate purpose of these studies 
the same. In Bishop's paper, we see the argument being made for data that 
support local decisionmaking. In particular, local policymakers and the 
citizens of a community need (1) data that will help them to understand bett< 
how schools influence learning and how effective schools develop and evolve, 
(2) comparative data on the performance of State and local educational systet 
so the citizenry and public officials can hold local adroinistrators accountal 

Data that will facilitate this type of local decisionmaking are available fr< 
number of existing and new sources. To support the firet of the two data ne< 
noted above, data from a number of sources would have to be merged (Bishop): 

o achievement tests for a great variety of subjects (not juet one 
subject at a time) which have separate subteets for higher order 
skills and basic ekills; 

o aptitude tests; 

o school records on courses taken, gradee, abeenteeisra, descriptions o 
special education services received, and test scores; 

o questionnaires measuring student background and attitudes (i.e., 
locus of control); 

o parent questionnaires; 

o eurveys of the students, teachers, guidance counselors, and 
principals; 

o multiple administrations of tests and surveys about 2 to 4 year3 
apart; 

o follow-up data on labor market outcomes with complete histories of 
employment and unemployment up to age 40; and 

o interviews with employers to measure skills and job performance of 
recent school leavers. 


The other need for data—i.e., to monitor the progress and achievements of 
State and local education systems—will require the following data (Bishop): 

o Performance on achievement tests in a variety of subjects that all 
(or almost all) students in the echool take; 


graaers witn performance 
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o Separate test scores for basic and high level skills; 

o State comparisons of the economic outcomes of schooling: earning, 
employment, occupstion; 

o Time series and regional data on salary offers to recipients of 
associate degrees by area of study; 

o Time series and regional data on the economic outcomes of schooling 
from an improved current Population Survey; 

o Time series and regional data on how schools influence the 
development of character (e.g. , locus of control); and 

o International comparative data on achievement in particular subjects 
and on time use. 


irge longitudinal data sets like NELSOO and core data that are comparable 
;ross States are required if the data are to be available and usod to impro 1 
lucation. 

te accountability issue--whether taxpayers are receiving what they should C< 
leir investment in public education—is an important one reiterated in a 
imber of papersMcClure and Plank describe a public investment cycle, wlii< 
».gins with taxpayers < or wage earners) who make an investment in education 
>r students) to produce wage earners (or taxpayers). education is thus a 
liddle product" in the investment cycle. Cost-benefit studies are needed t< 
jmonstrate the return on taxpayers' investments. 

.Cliire and Plank recommend that the federal government provide data bases 
lat track the investment cycle, since federal, as well as State policymakers 
ied this information to make productive educational investments. 

>ecifically, they suggest that NChS collect the following data: 

o Individual achievement and economic performance over time; 

o School site productivity; and 

o Regional economic returns on educational investments. 


rogram Evaluation . The recent interest in educational excellence and 
’fective schools has sparked a number of recommendations for program 
valuation: what makes a school effective; what about the school environmen 
Jlps to produce students with good work habits, self-discipline, 
ilf-efficiency, productivity in the workforce, and so on; how can schools 
isure even greater learning and achievement among their students; and, even 
>ro narrowly, what works, ha noted earlier in this chapter, many writers 


advocate greater research Into the effective schools issue. They want to 
learn more about school processes, climate, environment, culture, and 
structure. The context of the school has as much an effect on students as 
specific courses. Indeed, the idea that a given set of facts can be learned 
readily in different settings is now accepted almost universally. The next 
step is to identify those settings that promote learning. 

As noted in the previous sub-section, Bishop argues for collecting a range of 
data that will assist policymakers and local decisionmakers assess their 
schools' effectiveness. These include not only achievement data and data on 
expenditure per pupil in average daily attendance, but also longitudinal data 
on economic outcomes and data on how schools influence the development of 
character. Data for understanding and improving how schools provide 
instruction are also needed (McPartland). 

As noted earlier, several writers present models that they believe can assist 
NCES and other federal agencies in their efforts to design and implement a 
data collection and research program to answer some of these emerging questions 
McPartland also includes in his paper a proposed data collection instrument wit 
questions about the impact of school organization and claseroom instruction on 
student outcomes. This instrument, or one like it, might be used to measure 
between-classroora and in-classroora grouping practices, scheduling of students 
and teachers, and arrangements for teaming or clustering or instructional group 
(McPartland). 

More intensive research into the dynamics of student and teacher interaction, 
a vital component of school climate, is also important (Hawley). A model of 
learning productivity (Hawley) is offered tor identifying data related to the 
improvement of schools. The model relates educational goals to 
characteristics and quality of raw materials and students, the technology for 
producing learning, craftsmanship, and environmental conditions. 

Hersh also presents a model for school effectiveness, based on his own 
research on the topic. He contends that the following determine school 
effectiveness: 

o The people—teachers, administrators, and students; 

o The quality of effort, materials, and time; and 

o The curriculum. 


hese three factors are further delineated in the model, which presents two 
sets of attributes—social organization and instruction and 
curriculum-believed to be associated with effective schools. Which 

corabinat^nn^nf 3 under these tw ° headings are most important and which 
investiaation u° °u° leads to raoat e££ec£i -ve schools, however, requires mori 
calls iroInLlf f ** CaS6 StUdies to infinite what he 

a oarticn^i f 10 ? f a ° y ' altuation "here schools have attained both 

the lbUi^ t excellence in each of the attributes of the abiilty and 

the ability to improve continuously. 



occurs at the local and State levels, writers warn that the federal data 
gatherers must constantly keep this in mind and should understand and plan 
their data collection around State and local policymakers who want to be a 
to assess how their State or local school district ie doing in comparison 
other Statee and districts. The lack of comparability today, however, 
prevents these comparisons from being made with much meaning or precision. 

The problem of lack of comparability is pervasive in data gathering in 
general, but is especially critical in education. Virtually everyone poir 
this out in regard to one of the inputs or outcomes that must be measured. 
There are repeated pleas for common definitions, so many that it appears t 
if NCKS did nothing else but devise a system for collecting data in a way 
makes meaningful State-by-State comparisons possible, it will make many p< 
very happy (see for example, Buccino, Usdan, w. Turnbull, Bishop, and othe 

Yet, to compare two things, it is necessary for them to have some shared 
characterietics. This basic premise cannot be fulfilled with any certaint 
when comparing schools, however. The basic entities involved in the 
comparison are frequently very different. For example, the use of SAT see 
to compare achievement levels across schools or States is meaningless, 
if the social, racial, and economic backgrounds of the student populations 
not taken into account (Cronin, Bishop). Disparities similar to this occi 
virtually every aspect of education that is measured. This problem is 
especially significant when comparing educational inputs and outcomes. 

Bishop suggests that NCI2S work cooperatively with the States to insure the 
data are consistent with common definitions. If the specific data element 
are not comparable (i.e., if they are interpreted differently), they shouJ 
either not be published or published with a footnote describing reasons fc 
the lack of comparability. National aggregates should also only be report 
for the States that have provided comparable information. 



ISSUES ON DATA COLLECTION 
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irpose of thie chapter ie to identify the major methodological, 

Jlogical and technical issues for the collection of education data cited in 
»pers and synthesize the recommendations made. The specifications to the 
■8 did not explicitly request the authors to address these issues of data 
:tion. This fact, in eorae respect, has restricted the content coverage of 
;hapter. Nevertheless, several salient issues were raised in the papers as 
iced by the following outline. 

lological Issues 

Lternative Designs for Data Collection 
Sample vs. Universe Surveys 
Obse rva tional Studies 
Other Study Designs 

le Necessary Conditions for Comparability to Assess Differences 
d Changes 

The Range of Definitional Problems 

Some Solutions to the Definitional Problems 

Quality Control 

Aggregation of Data 

Consistency of Data as a Priority 

International Comparisons 

Public-Private School Comparisons and Non-School Educational 
Activities 

State-by-State Comparisons 
jlogical Issues of Data Collection 
ita Collection Technology 
itabase Creation and Organization 
ita Linkage 

>rms of Reporting and Modes of Dissemination 

Leal Issues in Collecting Data 

:curacy 

Lmeliness of Data Dissemination and Periodicity of Collection 


Jayless is a Vice President responsible for Educational Studies at Westat, 
Rockville, Maryland. 

Forbes is President of HUCO Associates, Wilmington, North Carolina, and 
:or of the Rural Education Institute at East Carolina University. 

Smith ie Senior Educational Researcher at Westat, Inc., Rockville, 

md. 

Womer is Professor of Education, at the University of Michigan, Ann Arbor, 
ran. 


The chapter is not a comprehensive collection o£ all of the ideas inclu 
the papers oTT^issues. Instead, the focus of this chapter ie to quo 
paraphrase the ideas that illustrate a number of technical data problem 
when resolved, would improve the quality of the data collection process 
appropriate, we infuse into the synthesis the relationship of the issue 
politics of data collection as suggested by the redesign authors. In c 
of this report, we will provide (in our professional judgment) what red 
NCES should take in its data collection programs to improve the quality 
Nation's educational data. 


METHODOLOGICAL ISSUES 
ALTERNATIVE DESIGNS FOR DATA COLLECTION 

Sample vs. Universe Surveys. The consensus of the writers who addrosse 
"how" of data collection favors the use of representative samples. Uni 
eurveys came in a distant second. 

Cronin's simple and straightforward comment, "I support a eimplified a a 
system of educational attainment," reflects the views of a large number 
writers. Cronin states that "data should be drawn from sample rathsr t 
populations. The information ordinarily will be just as useful and the 
data collection, especially to local educators, will be dramatically r€ 
Hill supports this view with his recommendation to reduce the "data ref 
burden by avoiding universal surveys." Harrison eupports the use of se 
when appropriate, but suggests the use of administrative record data ar 
universe surveys when necessary. 

The paper by the National Center for Research in Vocational Education c 
the appropriate use of samples in conducting surveys to provide data fc 
national policy purposes. Hilliard recommends the selective use of the 
approach. Implicit within his comments related to surveys of private & 
the recommendation for sampling the universe. 

Samples that provide State-level data ars implicit in most recommendatj 
Odden and Harrison are explicit in their recommendations about this toi 
Odden states that the sample ehould be "REPRESENTATIVE FOR EACH OF THE 
STATES, as well as for the nation as a whole." 

Observational Studies . Some writers comment on the collection of some 
data by using observational techniques rather than the mail surveys us< 
exclusively by NCES. For example, McPartland, Becker and Crain point ( 
"the more that questions deal with behavior patterns determined by 'cui 
'internal politics, 1 or 'general agreement, 1 and the more that the pati 
according to the characteristics of the specific instance, the more thi 
attention has to be paid to obtaining multiple sources of data about tl 
in question.’ 1 They suggest "measuring the behavior or policy at diffe: 
points during the school year, and using judgements of external observi 
than relying solely on self-reports of echool practitioners." Petersoi 
that data pertaining to how teachers allocate their time can be collec 
asking teachers to keep logs, but recommends that data "time so n and 


jperationalizing the qualities and behaviors that go into good teaching, samp 
>f teachers could be observed periodically to provide the needed data on oven 
ind special pedagogical attributes. 

)ther Study Designs . Some writers suggest that NCES use several other types 
lesigns to collect data in addition to the survey and observational designs 
previously reviewed. 

?he National Education Association (NEA) recommends that NCES consider expanc 
.ts "program to include case studies, field studies, policy reviews, (and) 
ilstorical research." A number of other papers support the use of a case-sti 
ipproach. Danner points out the importance of historical statistics in 
sstablishing "the context for the interpretation of long-term change" and the 
’perimeters of contemporary issues." Royston, writing for the U.S. Equal 
Employment Opportunity Commission, who also supports the case study design, 
suggests ethnographic studies "on how minority students 'move through' the 
3ystera, with emphasis on barriers and 'tracking.'" 

CHE NECESSARY CONDITIONS FOR COMPARABILITY TO ASSESS DIFFERENCES AND CHANGES 

falberg concisely states: "The value of statistical research depends on valj 
comparisons...." Selden reviews three ways that data can be used in conducts 
comparisons. The data may be compared with a standard, with data from anotht 
jroup, or with data from the past. All three forms of comparisons require 
iccurate data. This section synthesizes comments relating to the comparabili 
cf data. 


Che Range of Definitional Problems . A phrase by Cooke, Ginaburg and Smith 
captures the essence of the single issue raised most frequently relating to c 
compatibility: "Data can be reported incorrectly if definitions across 

respondents are inconsistent...." They provide as examples the problems ere* 
cy the lack of consistent definitions in determining dropout rates, student 
achievement, student victimization, class size, teacher salaries, and of cou: 
enrollments. 

Che paper by The National Center for Research in Vocational Education also 
provides examples of problems controlled by definitional inconsistencies. T1 
xamples include the lack of definitions for the following terras: vocational 
3tudents, disadvantaged and handicapped students, and program completers or 
Leavers. 

Berryman lists examples of inconsistent definitions in the collection of dat< 
pertaining to teacher supply and demand and data on education revenues and 
expenditures by categories. She states that "data such as these are usually 
comparable across states because the definitional variables differ." 

3arker, writing for the Rural Education Association (REA), suggests that the 
"lack of a precise definition may be one reason rural education has received 
ittle attention in recent years." For example, even the concept of "rural" 
definsd differently by the Bureau of the Census and some of the States. 


Hilliard states that "anyone who is even minimally familiar with schoo. 
aware that there is no common nomenclature Cor classes that would enab. 
meaningful analysis to take place regarding precisely what content is 

schools." 


Specific and implicit examples of definitional problems, whether conce; 
operational, are provided by many other writers. The following exarapl 
Lehnen in his submission on behalf of the American Statistical Associa 
illustrative: 


"Class size and Teacher Load Information: The current measures of 
class size reported in the Digest of Educational Statistics does n 
sufficient detail to be of much use. The averages reported for In 
example, in no way reflect the personal experiences of this author 
of teachers he has consulted. One general argument made locally i 
special education classes skew the class size distribution and dis 
mean, thus yiving the impression that class size is smaller than, 
exists." 


Some Solutions to the Definitional Problems. Some writers not only ra 
issue of inconsistent definitions, but provide suggestions for the col 
data from the administrative records of State education agencies. Dav 
suggests the use of a glossary that "should reflect" the many question 
asked and "indicate differences in definitions across states or data s 
Walberg recommends public accessibility of highly detailed, explicit 
descriptions of data definitions and collection procedures. 


Bishop states "that NCES needs to work cooperatively with the states t 
that data reported are consistent with the common definitions that hav 
adopted." The National Center for Research in Vocational Education ui 
stability in usage. Cooke, Ginsburg and Smith suggest that one of the 
developing a set of indicators would be a "move toward a common set of 
definitions across states...." 


Hilliard recommends that NCES "work with Chief State School Officers 1 
a common nomenclature for key academic courses" and to "collect and r€ 
based upon this nomenclature." w. Turnbull suggests "that NCES work 1 
Council of Chief State School Officers to procure comparable data fron 
Education Agencies." 

The paper submitted by the Council of Chief State School Officers incj 
following paragraph: 

"Standardization and coordination of data definitions at the fedej 
is a role that may be appropriate for NCES. This coordination, ai 
attendant acceptance of the development and distribution of gloss; 
is necessary to improved comparability of information. Additions; 
the acceptance of this role would increase the confidence of data 
that information in given formats would be available over time an< 
subject to changing program emphasis or approaches." 


expenditure per pupil reported by NCES for Indiana. He advocates that it sh< 
e possible to obtain such information according to a uniform reporting st&nc 
about every school district in the nation. He also goes on to point out thai 
there appears to be no technical publication reporting NCES operational 
efinitions, technical terms, data collection standards and practices, and tl 
quality control methods. 

arly in his Indiana school finance study, he suggested a reference librariar 
Indiana University attempt to obtain such documentation, which was done with< 
success. Subsequent calls to NCES in conversation with staff members reveal* 
that no such publication currently exists. He goes on to state: 

"The lack of such documentation makes it impossible to provide informati< 
about the interpretation of the statistical information. Furthermore, ii 
compromises the conclusions reached by the analysis using NCES data." 

The papers provide ample evidence of the need for consistent definitions and 
suggest that NCES should provide leadership in their development. Some 
additional comments on this topic by the authors of this chapter are include* 
Chapter H. 

Quality Control . A few papers suggest a need for quality control measures o\ 
the federally collected or accessed data {not limited to NCES). In this seci 
the terra quality control denotes that data are cross-checked or inspected 
between multiple sources. Although the topic of quality control is normally 
associated with validity and reliability issues, comments by writers suygesi 
is also appropriate to consider when discussing comparability of data. For 
example, David provides the outline of the needed quality control program: 

"Given the need to rely on data from other sources... NCES must, at the 
least, develop a system that permits crosschecking the data with other 
sources for the same information... .To the extent that multiple data sou; 
already exist, NCES should make comparisons across data sources and repoj 
on both the extent to which discrepancies are found and plausible 
explanations for the discrepancies.... Data for which multiple sources do 
currently exist should be collected through alternative means designed 
explicitly as a crosscheck." 

Lehnen, in discussing the Indiana public-use tape {produced by the Bureau of 
Census) that provides data on population and housing characteristics by scho< 
district, suggests that "the accuracy of the data needs to be verified.... 
Indiana public use tape contained numerous errors, including omission of 
districts and the combination of similar-named districts, and thus was 
unusable.... " 

The problem of using these data for comparing Indiana with other States is 
implicit in Lehnen's comments, as are the potential difficulties facing anal; 
in other States. 

Cooke, Ginsburg, and Smith, in discussing the variations between NCES and Bu: 
of the Census (Current Population Survey) dropout rate data, conclude that 


"until quality control studies are launched to explore the Census and 
numbers in detail, however, the true national dropout rate will reraair 
mystery." Their comments underline the difficulty of comparing dropoi 
from these two sources. They also recommend that a "system of qualit> 
that would catch...glaring statistical errors" be developed. 


Plisko, Ginsburg and Chaikind compliment the High School and Beyond (I 
quality control efforts pertaining to student-reported courses and gr; 
they indicate the need for additional quality control efforts within t 
program. They recommend the establishment of an Office of Quality Cot 
scope of the projects this office would inspect for lack of quality ws 
specified. 


Aggregation of Data . Comparability issues related to the aggregation 
are raised four ways by the writers of the papers, as follows: 


(1) Variations in statistics within reporting categories; 

(2) Variations in the sources from which the data are collected; 

(3) Willingness of agencies to participate in the data collection pr< 
and 

(4) The appropriate unit to aggregate the data. 

The first concern focuses on data related to variations in statistics 
reporting categories. Hilliard identified this issue in discussing tl 
effective school research movement: 

"For the most part, isolated schools that were "swimming upstream' 
buried in aggregated data which tended to suggest that no such scl 
existed. In fact, analyses of much of the school effectiveness r< 
to the erroneous conclusion that schools had little or no effect. 
Questions such as "Do schools work?" were common. It is notable 
following the effective school research, the question more often . 
good schools work? 1 The same may be said of effective teacher rei 

Hilliard underlines this point by stating that "data aggregated at thi 
school district level may serve some useful purposes but, for many pu: 
most significant information is the presence or absence of a pattern ■ 

variation among school sites or even among school classrooms, sometinn 
given school site." 


Bishop, Valdivieso and others provide implicit examples. Thomas, disi 
need for more "consistency and coordination among data collection age: 
efinition, aggregation, and disaggregation of minority groups... 


Variation information lost through aggregation 
data and creates comparability concerns about 
topic, but use different aggregates of data. 


reduces the usefulness 
studies that focus on t 
Hilliard recommends tha 


(and) disaggregated data." 


he second comparability concern related to data aggregation is identified by 
Ehe National Center for Research in Vocational Education. They point out tha 

"A...major problem the old VEDS (Vocational Education Data System) 
encountered was the varied and decentralized nature of the system that 
generated the VEDS reports. The VEDS forms were distributed to the state 
The information that was aggregated and reported on those forms was 
collected from local educational agencies by a variety of means. A few 
states virtually duplicated the VEDS forms and required the local agoncie 
to complete them. Some states relied on individual student records which 
were completed at the local level and aggregated at the state level. Mos 
states, however, tried to adopt their existing information systems to 
supply the information required by VEDS. The success of this approach 
varied widely across states." 

The inference drawn from these comments is that the aggregate data are only a 
good as the source data. If the source data are not comparable then the 
sefulness of the aggregate data is questionable. 

The third issue related to aggregation pertains to the willingness of local a 
state education agencies to participate in data collection efforts. Hill 
reviews three "factors that lead state and local agency officials to resist 
federal data collection efforts or provide low-quality responses’* which lead 
data comparability problems. The three factors Hill discusses are 
administrative burden, federal presuraptuousness, and fear of harm. The last 
factor is discussed in terms of avoiding enforcement actions and avoiding 
embarrassment. 

Hill offers six suggestions for reducing "the severity of state and local 
resistance:" 

(1) "That school districts will resist federal data requests less if they sc 
fewer of them" and that "complaints against (the) federal data burden could t 
significantly reduced by a greater use of sample surveys." 

(2) That because individual research firms are viewed as being more "interes 
in doing research" than in "compliance reviews" and because individual reseat 
firms "can build reputations for fairness and professionalism," they can "get 
far better cooperation" and results "for the smaller sample surveys and 
exploratory studies." He suggests that NCES "make greater use of contractors 
collect data." 

(3) That "school systems will contribute more willingly... if they expect to 
benefit directly from the results." Providing documents that smaller distric 
"could use in reports to their own school boards and the public" are actions 
that could reduce resistence. 

(4) That members of Congress tend to support their constituents in conflicts 
pertaining to data collection. He suggests that "the best way to reduce 


makes "the value of the data collection errors avwwu ^ 
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(5) That "OCR's {Office of Civil Rights) school district surveys are « 
problem for NCES." He suggests that negotiating "with OCR to reduce t, 
demands" would reduce resistence. 

{6) That the way to improve cooperation la to work individually rathe, 
with groups of Chief State School Officers in negotiating the resolutii 
objections. "As indviduals," he reports, "Chiefs generally have a bro. 
perspective and are eager to cooperate in studies that might illuminat 
important national issues... {with) a good explanation of the study's 
importance." 

The fourth concern related to data aggregation is covered by Danner in 
following quote: 

"In many cases, data are unwisely aggregated or aggregated in form 
reduce their usefulness. Despite jurisdictional realities, a bett 
for comparison of much data is probably the standard metropolitan 
rather than the state, at least for city schools. Analogously, da 
regarding public and private schools should be distinguished. Muc 
data fail to reflect the dual system of American education at all 
and even those few efforts to distinguish between differsnt kinds 
private and religious schools are not carried out consistently in 
now published." 


There is also a conceptual or substantive aspect of the aggregation pr 
needs to be recognized but is not further dealt with here. It relates 
appropriate unit{s) of analysis for statistics to support the educatic 
movement — the classroom, the school, the school district, and the St 

We agree that the issue of aggregation is difficult to deal with in th 
abstract. The appropriate level of aggregation depends on the purpose 
of the data being collected and analyzed. 

Consistency of Data as a Priority . Berryman in reviewing data pertair 
current and projected teacher supply and demand information makes the 
observation about the consistency of data over studies and time. 

"We have data, but from different sampling frames, differently woi 
questions, and different time periods. These non-comparabilities 
impossible to pool available data to increase the statistical prec 
'snapshot' (cross-sectional) estimates or to create a time series. 

Banner emphatically makes the same observation: 

Moreover, the data that is gathered and published must be consisl 
time. Too often, the existing data series are presented different: 
oth to altered data gathering methods and changed survey questior 


evaluate changes in educational conditions over time. 


lawley comments on the "variation in the information collected from state to 
state" as one of the difficulties of data that "have potentially significant 
lses in fostering school improvement.... 

Dhis same theme occurs in Smith's paper which describes "the lack of 
correspondence of the High School and Beyond (HSB) tests with the IEA 
[International Education Assessment), the National Assessment, or state 
assessments." He observes that "there seems to be, at best, scattered 
coordination. ..." 

falberg points out that "the performance of third and sixth grade students" 
cannot be compared "because comparisons are strictly valid for only students ’ 
lave taken the same test...." 

Chese concerns related to comparability over studies and time are implicit in 
statements and suggestions provided by many other writers. For example, the 
'Jational Education Association comments that "the use of the same questions f 
cublic and private school surveys permits more extensive matching and compari 
cf school systems on a wide variety of attributes." 

'Jatriello discusses consistency between national surveys and the work of 
Individual researchers: 

"In the foreseeable future it is likely that only the federal government 
will be able to mount educational research projects involving the collect 
of large nationally representative data sets. Vet many new and interestin 
theoretical ideas and most richly textured studies of educational 
phenomena are developed by individual investigators or small teams of 
investigators working in a small number of schools with severely limited 
research budgets. If the large-scale data collection efforts of NCES are 
to profit from and enrich the work of most educational researchers, NCES 
will have to put in place a process to ensure that linkages can be made 
between its macro-level data collection program and micro-level 
investigations. 

tfalberg follows his earlier observation about the non-coraparability of third 
3ixth grade tests by suggesting the calibration of "items and tests to nation 
3tandard tests." He suggests that an expanded National Assessment could play 
that role. He also suggests that computerized "tailored-testing" may offer a 
solution. 

Smith calls for someone to "systematically set out the interrelationships amo 
he existing surveys and examine the opportunities in the future so that maxi 
coordination—could be achieved." W. Turnbull states that a "system of plann 
•linking sections' common to different data bases might prove feasible and 
helpful." Hawley suggests that NCES "'nest' future studies in such a way tha 
data from the same sites could be integrated." 


Tnf.flrnational Comparisons. Information that provides for international 
comparisons is often' iUEad by writers addressing the question o£ wha, 
should be collected. Several writers provide comments and suggestions 
to the "how" of collecting international data. 


smith describes the state of international data and suggests a first si 
resolving some of the problems. 

"The quality of data comparing the resources, organization, intent! 
outcomes of the various advanced nations of the world is very poor, 
{International Educational Assessment) IEA surveys, which tragical, 
state of the art in this area, suffer from a Lack of connection wii 
of the established ways of insuring adequate data collection. This 
combination of problems has led to erratic schedules for data coll* 
very poor response rates for U.S. samples, and a lack of use of th< 
by the U.S. policy system. One way of beginning to ameliorate thes 
problems would be to have NCES assume responsibility for coordinat. 
involvement in IEA activities." 


Murnane also addresses the problems associated with international corap< 
He states that: 


"...differences in the quality of national school systems is only « 
many reasons why average test scores differ amony countries. Cons' 
I am skeptical about the possibilities of drawing reliable inferen* 
U.S. education from international comparisons at a single point in 
Comparisons over time offer much better prospects however. In par 
it is possible to examine how the achievement of U.S. students, as 
on the IEA tests, changes over time, and whether the position of U 
students relative to students in other countries changes over time 

Murnane also suggests that "the Council of Chief State School Officers 
involved in administering the tests." 

Public/Private School Comparisons and Non-School Educational Activitie, 
writers call for the collection of private school data to permit a com 
understanding of the private education system. Some are explicit in 
recommending that data collected from and about private schools be com 
with public school data. For example, Grant in discussing private sch 
suggests that "the data should be consistent from one year to the next 
should be comparable with the figures we obtain for public schools." 
National Education Association recommends the alignment of data in sur 
public and private schools "so that the two surveys are comparable." 
in discussing the comparability of public and private school students 
recommends: "As much as possible collect the same data from private s 
from public schools." 

Moser writing for The Lutheran Church—Missouri Synod, also calls for 
ata for public and private schools. He states that "it is also helpf 
f we can separate The Lutheran Church— Missouri Synod schools from th 
chools in your non-public school survev. a d s. An . nA v.* 
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kinds of private and religious schools...." 

Implicit in Murnane's discussion of for-profit schools is a request for 
comparable data. He states that it "would be worthwhile to learn more about 
number of for-profit elementary and secondary schools in the U.S." Once we . 
the answers to these questions, we could explore whether for-profit schools 
operate differently fron not-for-profit schools." The collection of compara' 
data is also implicit in Usdan's recommendation that data " on the educationa 
process and ways of improving student achievement should be gathered from al 
deliverers of education services and not just from traditional schools servii 
to 17 year-olds. " 

Natriello, in discussing the need for collecting data on schooling that go 
beyond traditional schools, provides the following comments: 

"Unlike public and private full-time day schools, non-school educational 
resources may be difficult to identify. Several strategies may be neces 
to develop data on these education activities. First, it would be usefu 
include items on regular NCES surveys of students and parents.. .which 
request respondents to report on the extent of their participation in 
non-school educational activities.... 

"A second strategy for collecting information of non-school educational 
activities would be to identify the population of educational service 
providers through state corporate records.... 

"A third strategy for collecting information on the extent on non-school 
educational activities would be to identify a representative sample of 
communities and survey the available non-school educational programs 
available in the community." 

Stafre-by-State Comparisons . Comparable State level data are recommended by 
writers. 

Cooke, Ginsburg and Smith report that in state per-pupil expenditures the 
year-to-year variations of 30 percent or more make the data "extremely suspe 
They also support the Chief State School Officers' "recommendations designed 
improve and standardize the collection of data {including achievement test d 
at the state level." 

Berryman also points out comparability problems with state level data. She 
attributes most of the problems to "different definitions" that make the dat 
"usually non-comparable across states." 

Bishop reviews political concerns relating to interpretations of State-by-St 
comparisons that do not take "into account the demographic background of the 
students...." He states that "comparative data that purport to measure the 
performance of an educational system but in fact measure the talents and 
background of the students could confuse the public debate on education more 



consideration: 


" Changes in the relative performance of particular cohorts of child 
particular states . Such a statistic can be constructed by putting 
from testa administered in the first and later grades on a common m 
(e.g., standard deviation units, grade equivalents or state ranking 
then examining how the state performance on this ranking changes as 
level increases. While such comparisons might be made from differe 
given at a point in time, comparisons would be more valid if the co 
held constant. This would be done by calculating state rankings on 
grade scores in 1976 and then comparing them to the rankings on 8th 
scores in 19S4. Grade equivalents and standard deviation units wil 
different results. If grade equivalents are to be reported, standa 
deviation units should be reported as well and the difference betw 
two should be explained. 

" Achievement test scores {levels and gain scores) that have been ad 
for the demographic composition of the states 1 students . Another w 
reducing the bias problem discussed above is to estimate statistics 
predicting achievement as a function of student background characte 
using state aggregate data and then report only the residuals from 
model. *' 

Cronin rejects use of SAT measures as a comparative state education ind 
because of the demographic differences among students taking the SAT. 

Humane reviews comparability issues associated with dropout data. He 
recommends that "NCES should work with the Council of Chief State Schoc 
Officers (CCSSO) to develop and implement a uniform methodology for cal 
dropout rates" and that "NCES should encourage, and if possible, fund s 
that examine whether dropout rates as calculated by applying a new unif 
methodology to school, school district, and state level data are close 
dropout rates calculated from longitudinal data on individual students, 
that provided by High School and Beyond (HS&B)." 

Nurnane also reviews the difficulties of obtaining comparable teacher s 
data. He makes the following recommendation: 

"The federal government should publish on an annual basis comparisc 
salaries in teaching with those in other occupations. The comparis 
should be presented separately for each academic field. Useful con 
would be starting salaries, and salaries for individuals with ten > 
work experience. Data on. starting salaries are collected currentl) 
Placement Center of Northwestern University, and are published by 1 
National Educational Association. Consequently, it may not be nect 
NCES to do all cf the data collection. In fact, it may be efficier 
contract with Northwestern to collect comparable salary data for e> 
workers. However accomplished, it is important that annual data bt 
available to assess trends in the salaries of beginning teachers at 
experienced teachers relative to salaries in other occupations." 



Level Data Bases Related to Significant Educational Issues .” He states that one 
advantage to this approach would be some assurance "that the data collection was 
done properly." Implicit in his recommendation is the need for comparable state 
level data. 

The Council of Chief State School Officers' paper points to the costs of good 
data: "accurate and reliable information at the federal level is possible in 
direct proportion and relationship to the development and improvement of support 
systems at the state and local level." The Council "encourages NCES to seek 
sufficient resources to permit all levels of government to have resources to 
generate the data that the Federal Government needs to report timely, accurate 
and comprehensive statistics." 

Cronin discusses the cost effectiveness of collecting NAEP data. He suggests 
that "it is quite possible that assessment activity by a coalition of state 
educational agencies can be more cost-effective, more useful and more 
comprehensive than NAEP at present." 

B. Turnbull suggests that "NCES should not hesitate to take a strong stand with 
State Education Agency's (SEAs) on quality and consistency in the data they 
provide to the federal level." This view is supported by Plisko, Gineburg and 
Chaikind who recommend that "special attention must be given to holding the 
States accountable for providing the Department with consistent and accurate 
data." 


TECHNOLOGICAL ISSUES OF DATA COLLECTION 

Several of the suggestions and concerns expressed by the authors of these papers 
have some implications for the technology of collecting and processing data. 
However, this section is a synthesis limited to explicit references to 
technological issues. 

Technological issues are defined as those involving the hardware and software 
used in the data collection and data processing activities of NCES, whether 
directly performed by NCES or by contractors. The organization and structure ol 
databases (e.g., distributed vs. centralized database, integrated vs. separate 
files, etc.) are included in this definition of technological issues, but the 
content of the databases (i.e., which data items are in the data base) is not 
included. 

No single paper was dedicated to the theme of data collection and/or data 
processing technology. This is probably good, since technology should always b< 
determined by the larger substantive issues. But there is some disadvantage in 
not systematically addressing technological matters when current systems are 
already in place. The costs and benefits of more fully utilizing available 
technology, or of adopting other technologies, can become important 
considerations in these circumstances. 

Four major areas of concern about technology have been identified in the papers 


o 


Data collection, 





o Database creation and organization, 

o Oata linkage, and 

o Forms of reporting and modes of dissemination. 


Some authors make only cursory general comments like "we need more time 
dissemination of data" while other authors get very specific. Usually, 
only refer to technology in the context of some particular substantive 

Data Collection Technology . The fact that NCES places heavy reliance u 
surveys was noted. Plisko, Ginsburg, and Chaikind suggested that great 
consideration be given to telephone surveys and to the possible use of 
networks. The latter would be most appropriate where the number of res 
is limited, such as the 50 states. The main reason these alternative 
technologies were mentioned was to improve the timeliness of the data. 

In its letter response, the National School Boards Association (NSBA) n 
that new electronic technologies used in the schools for management anc 
instruction sometimes "enhance the ability of school management to gatV 
data about the success of the schooling enterprise." Although NSBA die 
mention this as an aid to NCES data collection efforts, it seems close! 
related. 

Two authors specifically mentioned the developing technology of compute 
assisted student testing. Walberg drew the analogy between giving the i 
to a whole class of students to "what would be called 'batch processing 
industry." He noted that the computer offers the possibility of "tail< 
testing" which is more efficient in terms of the amount of information 
be collected with a Limited number of items. Along these lines, Smith 
that computer is being explored for creating "testing environments thai 
more than the basic skills" such as critical thinking and higher order 
skills than can be measured in multiple choice and similar formats. 

Database Creation and Organization . Few comments were made concerning 
issues of database creation and organization. (Instead, most authors 
database content.) However, two papers specifically suggested that NCi 
consider a distributed database system. McClure and Plank suggested t 
consider instituting regional databases. Natriello suggested a pilot 
develop state-level databases which could then be evaluated to "select 
successful data base design and use it as the model for a national dat 
assembled from data collected by individual states." 

Data Linkage . The idea of "data linkage" is often referred to, at lea 
implicitly, in the papers. There are three fairly specific ways in whi 
idea appears, namely: 

o The "linkage" of data collection efforts in the sense of consolid 
reorganizing surveys; 


items in different surveys, in various non-survey databases 
(administrative records), or in the same survey over time; and 

o The "linkage" of data from different sources, such as from surveys and 
various other federal databases, to obtain more complete information us. 
existing data. 

Data linkage in the first sense is referred to by Hawley who discusses the 
"integration and enhancement of existing data" and recommends the nesting of 
"future studies in such a way that data from the same sites could be 
integrated." He suggests that "it might be possible to conduct the High Sch< 
and Beyond (HSB) and National Assessment studies in the same or overlapping 
locations." He also suggests the connection of school process studies "in s< 
way to the outcomes being studied in the NAEP." 

Also, with reference to data linkage in the first sense, Plisko, Ginsburg an< 
Chaikind make the following observation. "In particular, we need to examine 
whether the division between repeated cross-sectional studies, such as NAEP, 
longitudinal studies, such as HSB are real or artificial distinctions." The; 
outline the possibility of attaching a small longitudinal component to the NJ 
7th grade sample and to conduct a follow-up in two years to provide "some 
measure of the extent of attrition at this early level." 

Data linkage in the second sense involves comparability that is discussed 
elsewhere in this paper and will not be discussed further here. 

Data linkage in the third sense is often quite feasible but presents a numbe; 
technological problems in database organization, algorithm design for record 
matching, and data processing efficiency. 

Data linkage in the third sense is especially relevant to Federal student 
financial aid programs. As Plisko, Ginsburg and Chaikind put it: 

"Documenting the trends in aid recipients...has proved difficult and 
controversial. However, the problem is not that there is insufficient 
information, but that different data sets are disjointed and cannot desc: 
the total student aid package in relation to student financial need. Hei 
this problem is one of linkage rather than coverage...." It is worth not 
that High School and Beyond data are currently being linked to Federal 
Guaranteed Student Loan (GSL) and Pell files so that there are signifies: 
developments in technological capabilities for data linkage currently 
underway. Nevertheless, there are other sources of data which could be 
considered for linkage purposes and it is still true that "much of the 
information useful for analyzing the higher education process is pieceme. 
serving only the specific needs of the originating agency." 

While further development of linkage technology will not redress this entire 
it has the potential for producing major benefits. 

In a similar vein, W. Turnbull refers to "merging" various datasets, perhaps 
using a system of planned "linking sections." He seems to be referring to 


several possibilities here, involving data comparability, a common cor. 
and linkage in the sense being discussed here. 

Forms of R eporting and Modes of Dissemination . Data collection is a t. 
consuming process, and the issue of the timeliness of data is a critic, 
will be addressed in a later section of this chapter. The need for raor* 
dissemination of data is mentioned by the NEA, Plisko, Ginsburg, and C] 
and Coleman. While all of the problems associated with timely data 
dissemination are not technological, many of them are. The cumulative 
small delays in the many data processing steps involved in file creati' 
easily cumulate into major delays in making data available. 

The "level" of data which should be released is an issue which deserve; 
consideration. The NEA expressed a need for "raw" data, meaning data 
they could perform their own analyses. At the same time, text and tab 
produced by NCES might be distributed in electronic (e.g., floppy disk 
suggested by Miller. 

The quality of released data in terms of how much editing and cleaning 
performs on the data was of some concern to the NEA. Also, the need f 
accurate technical documentation, such as record layouts, was stressed 
NEA and Hawley. But of even greater concern was the need for NCES to ; 
data in formats that are readily usable, such as Statistical Analysis 
files or other "end user" formats, including such media as floppy disk 
Hill; Lehnen). 

As users become more technically sophisticated there is naturally more 
for immediate online access to data. This growing user sophistication 
particularly evident in the papers by Natriello, Plisko, Ginsburg, and 
and Coleman. It is worth noting that the Gutman Library at Harvard is 
its federally funded survey database on microcomputer use in schools o 
COMPUSERVE network for easy access. 

It is not necessary to move to direct computer network access for NCES 
to realize significant improvements in data dissemination. Lehnen put 
the suggestion the DIGEST ought to be issued on tape similar to the we 
the Census Bureau issues the City and county Data Books on tape. This 
suggestion has the merit of more fully utilizing readily available tec 
implement a dissemination strategy that has been proven to be highly s 


TECHNICAL ISSUES IN COLLECTING DATA 

This section synthesizes the technical issues of accuracy and timelin< 
collecting data. The approach is to illustrate the defects or noncon] 
the products of NCES data collection processes cited in the papers, 
chapter (where we are free to give our own comments), we advocate thai 
solution to these data problems is to improve the quality of the data 
process. 


and/or concepts have been combined into one construct which we term “accuracy, 
he four characteristics are validity, reliability, sampling, and completeness 
Survey sampling statisticians have labeled validity, reliability, and 
completeness as non-sampling errors. 

David expresses the importance and priority of data accuracy very well: 

“Although it is certainly important to drive data collection by the 
questions of interest, IN THE CASE OF NCES DATA ACCURACY IS BY FAR THE M0£ 
CRITICAL ISSUE. IF THE DATA CONTINUE TO BE AS INACCURATE IN THE FUTURE AS 
THEY HAVE BEEN IN THE PAST, ALL OTHER ISSUES ARE MOOT. Careful choices 
about what data to collect and clear reporting and interpretation cannot 
compensate for inaccurate data.' 1 (Emphasis added.) 

Contrast David's views with those of Grant who raises the issue of the trade-c 
etween accuracy and timeliness; 

“Statistics...(on)...public school enrollment, attendance, teachers, 
graduates, revenues and expenditures, should be collected on an annual 
basis. IN PREPARING THE REPORTS WE SHOULD EMPHASIZE SPEED RATHER THAN 
PRECISION, SO THAT THE DATA CAN BE PUBLISHED BEFORE THE END OF THE SCHOOL 
YEAR FOR WHICH THEY RELATE. This means that the financial data in the Fa! 
report will be estimates rather than the final, audited figures. When the 
Fall survey is repeated, the respondent should be encouraged to report an) 
changes that have occurred in the data they submitted for the previous 
year, and those corrections should be printed in at least one subsequent 
edition of the publication." (Emphasis added.) 

By this methodology, Grant feels timely school statistics will be published 
annually. 

Yet, Plisko, Ginsburg and Chaikind raise another issue that suggests efforts t 
improve the quality of educational data are contrary to accuracy from a 
political perspective. 

“The tremendous national interest in educational improvement provides 
impetus to push for reform of our national base of educational statistics. 
But reforms will be not easily accomplished. Representatives of some 
educational special interest may not want improved statistical informatior 
for fear their activities will be placed in an unfavorable 
light.... Moreover, many in Congress do not give educational statistics s 
very high priority. Political benefits come from providing direct service 
to constituents, not better data." 

It is clear that for NCES to redesign and improve the quality of its data 
collection processes, trade-offs are going to be needed between cost, technic* 
issues (accuracy), timeliness, and the political considerations. 

Plisko, Ginsburg, and Chaikind suggest that the current NCES elementary and 
secondary data have certain accuracy problems and recommend improvements in t) 
data for special needs populations, class size, home learning, and school 
discipline. 



To improve the overall quality of NCES data collection processes, SmJ 
recommends three areas, that, in his view would improve the coverage 
accuracy of NCES data. Specifically, his recommendations, which are 
that of Plisko, Ginsberg and Chaikind, are: 


M > An external group or internal NCES staff be used to recomm, 
data should no longer be collected. 

(2} A system of yearly internal and external reviewe of the dai 
collection process (method used for collection, and method 
analysis and reporting) be established. 

(3) An ongoing system for partial verification to cross-valida 
response quality and/or frame coverage of NCES surveys be < 


Timeliness of Data Dissemination and Periodicity of Collection . In . 
the quality of data, timeliness -- reporting data promptly — is an 
discussed by several of the authors. A major criticism is that cert 
data are reported too late to be useful for policymaking . In addit 
issue of the periodicity, that is, the frequency in which the data a 
(annually or biannually) is mentioned as an important issue. 

Plisko, Ginsburg, and Chaikind point out several problems that relat 
timeliness of data. In particular, they suggest the problem associa 
NCES enrollment data of elementary and secondary schools of the Comm 
(CCD) is that the most recent data available is for the 1982-83 scho 
They feel the data can be collected and processed sooner since a pri 
has developed and published enrollment data for the 1984-85 school y 
provided more information on each school building than the CCD repor 
claim that a major reason for this is that the private firm uses the 
for data collection purposes, where the NCES/State approach is depen 
mail questionnaire involving the States, school districts and school 
They also point out that the data that NCES publishes on finances an 
of the schools is at least 1 or 2 years older than like data publish 
National Education Association. 

Plisko, Ginsburg, and Chaikind, further comment that once data are c 
the reporting process should not languish. They illustrate this frc 
survey of teacher demand and shortages by stating that the inordinat 
time between final data collection and preliminary disseramination of 
inappropriate. They point out that if NCES has insufficient staff 1 
the data collected, analysis contracts should be built into the ovei 
collection effort. 

Cronin, in his paper, recommends a specific standard relative to the 
of educational data. 

The amount of (educational) data and number of indicators shou: 
limited to that which can be stored and analyzed within three m< 
reported to policymakers within the year." 


population surveys would improve the timeliness of the data. The major 
justification of his recommendation is that information from sample surveys v 
ordinarily be just as useful, and the cost of data collection especially to 
local educators will be drastically reduced and timeliness of the data improv 
An implication of Cronin's recommendation is that the educational statistics 
will need to be interpreted with their associated sampling errors, which 
requires a minimal level of statistical competencies on the part of the 
publishers and users of the statistics. 

Cronin also articulates a standard relative to the timeliness of reporting 
student performance results: 

"A report is useful to decision makers if the (student performance) resul 
are available within 6 to 10 weeks — such as the College Board and ETS c 
provide to college admission officials." 

Cronin offers an additional timeliness standard: 

■’The speed of analysis in reporting (data) must be timed to the decision¬ 
making cycle of the planning/budgeting cycles of the states, which vary, 
to a federal reauthorization or budget issues." 

He strongly points out the need for data to be current. 

Reece, writing for the Association of American Publishers, points out that 
summaries of course offerings, enrollments, and curriculum practices in the 
public secondary schools are critical data needed to estimate market size anc 
other vital factors for the educational publishing industry. He points out t 
the publication by NCES of such data every 10 years is clearly not frequent 
enough, primarily because of the dramatic changes that occur in the curricula 
in our public school system over a decade. He advocates a change in the time 
for the collection and publishing of such data. He goes on to point out that 
sufficiently reliable data can be collected through probability samples at a 
reasonable expenditure by the government and enrollment data should cover gra 
7 and 8 as well as grades 9 through 12. 

Berry, of the National Science Foundation, points out that the most signifies 
determinant of teacher demand projection is turnover rates which are age 
specific. Yet when NCES data on teacher turnover rates were last collected 1 
the 1969-70 school year even these earlier data are not age specific. 

Barro suggests that State-by-State data are reported in the Condition of 
Education and Digest of Education Statistics typically with lags of 3 to 4 
years. Financial data for the selected large local districts have been 
published irregularly in the past, but the latest such data to appear in the 
Digest of Education Statistics are for the 1979-80 school year. Barro goes < 
to point out the consequences to policy analysts of not having the diatribut: 
of revenue or expenditure among local school districts. Such distributions 
within States, in his view, have long been the central concern of school fin< 
policymakers and researchers. Not having such data in a timely fashion is a 
weakness of NCES data. 


SUMMARY 


As indicated in the introduction of this chapter, writers were not expl; 
asked to address the "hows" of data collection. This chapter synthesiz< 
remarks that were provided. It does not attempt to include concerns re. 
data collections that were not raised by the writers. The omission of < 
should not be interpreted as an indication of their importance or lack 


"The National Center for Education Statistics plays an important 
role for the federal government -- collecting statistics useful for 
local, state and federal decisionmaking about education” (Cronin). 

entral role of the Federal government in the acquisition of data about 
ation's education system dates from the establishment of an education 
traent in the Federal government by Congressional edict in 1667: 

"That there be established, at the city of Washington, a Department 
of Education, for the purpose of collecting such statistics and 
facts as shall show the condition and progress of education in the 
several States and Territories, and of diffusing such information 
respecting the organization and management of the school systems, 
and methods of teaching, as shall aid the people of the United 
States in the establishment and maintenance of efficient school 
systems." 

service to the reader, and as a reference point for considering the 
us suggestions of the writers as presented in this Chapter, the current 
on of the NCES is 

"The purpose of the Center shall be to collect and disseminate 
statistics and other data related to education in the United States 
and in other nations. The Center shall — 

(1) collect, collate, and, from time to time, report full and 
complete statistics on the condition of education in the 
United States; 

(2) conduct and publish reports on specialized analyses of the 
meaning and significance of such statistics; 

(3) assist State and local education agencies, including State 
agencies responsible for postsecondary education, in improv- 
ing and automating their statistical and data collection 
activities; 

(4) review and report on educational activities in foreign 
countries; and 

(5) conduct a continuing survey of institutions of higher education 
and local education agencies to determine the demand for, and 
the availability of, qualified teachers and administrative 
personnel, especially in critical areas within education which 
are developing or are likely to develop, and assess the extent 
to which programs administered in the Education Division are 
helping to meet the needs identified as a result of such 
continuing survey" (Section 406(b), General Education Provisions 
Act, as amended (20 U.S.C. 1221e-1)). 


:. Taeuber is Research Director, Division of Elementary and Secondary 
lucation Statistics, National Center for Education Statistics, and 
^-Director of the REDESIGN Project. 


Many of the contributors to this Redesign Project nave suggested ways 
NCES could, and should, modify the interpretation of its mission beyoi 
current set of activities in order to better support its diverse data 
the education community and throughout the public at large. Hawley oi 
suggestion for Federal government coordination, with the responsibilii 
by the Secretary of Education: 

"The Secretary of Education could take the initiative in de: 
a master plan for education-related statistics that would et 
the statistical activities of all of the (federal) agencies 
first step in that regard would be to catalog current and p 
programs. The second step would be to identify the key var 
upon which major studies focus and the uses to which data a: 
part. ... An interagency effort could (a) identify sources < 
(b) suggest how existing data can be integrated, (c) identiJ 
of unnecessary hindrance and important issues about which d< 
needed, and (d) provide advice to the developers of major ni 
efforts to collect educational information." 

Broadening beyond just the federal agencies, Barro suggests "that MCE: 
the lead in exploring with other agencies, public and private, the fe< 
of achieving greater compatibility among data sets." 

These and other comments suggest that writers see a need for A NATION; 
with coordinative and oversight responsibility, and authority, over tl 
domain of data acquisition by the federal government from the educate 
of the Nation. Some writers see this as a coordination, not executioi 
charge; as Hawley comments: 

"This is not to argue that all data collection efforts with 
Department of Education should march to the same drummer, 
point is that an enormous amount of information is collectei 
American education but there is no central effort to plan o: 
coordinate the information collected or even to consolidate 
it is collected.’’ 

In a recent letter to this Project, the staff of the Council of Chief 
School Officers asserts that the coordination role defines a "true ce: 
education statistics." Their full recommendation follows: 

"We strongly urge that the function be a true statistical c 
that assumes the major responsibility for coordination of t 
collection, assembly, analysis and dissemination for that S' 
society under its purview, namely education. 


The Secretary of Education would be required to make a cle 
committed designation that the Center would have responsibi 
coordination of statistical data collection and analysis ac 
across the Department of Education regardless of organizati 
, ne ® an ' or bureaucracies. This assignment would also req 
e enter be charged with promoting the integration of the 
ata collection activities conducted by other federal agenc 
t °*- Agriculture, Bureau of the Census, Departraen 

a r, et al) and related private agencies (National Educat 
Association, American Cou nil An A« A ^ J A.L.A. ___ J 


in NCES (e.g., CCD, VEDS, NELS-88); 2) expansion of the system to 
include those other data collection activities by the Department of 
Education (e.g., Special Education, Chapter I of ECIA, Chapter II of 
the Math and Science Act); and finally 3) establishment of out-reach 
activities to other agencies to ensure appropriate federal and 
national coordination. Included in this function would be defining 
a common set of data elements across the spectrum, coordinating 
collection of all statistical data, developing efficient collection 
and dissemination systems (in conjunction with users and providers), 
seeking out current needs for educational information, and providing 
assistance, both technical and financial, to the respondees and 
users of educational data. 

"Any effort at a ten-year plan, without a clear understanding of the 
agency’s mission and philosophy, offers little promise of success. 
Additionally, in our view, the failure to expand the mission and 
functional boundaries of the National Center to a true center for 
education statistics limits the potential growth to little more than 
that capacity which exists today." 


STATE DATA -- ACQUISITION AND DISSEMINATION 

itically speaking, the primacy of the States in education matters means 
lust be available on a State-by-State basis. As Lehnen states with 
:t to the national data of NCES and others; 

"National averages and other statistics do not reveal much about 
the state education systems ... Yet it is the states who will 
determine the direction and scope of education policy and not the 
federal government. Without this detail NCES data will have only 
limited utility for policy studies within states." 

itional Governors’ Association (NGA) adds; 

"In order to perform education policy setting functions, states 
need to plan, develop, implement and evaluate education initiatives. 
... national trend data and consistent and accurate data from all 
states for macro comparison purposes is of key interest ... samples 
should be examined to determine the feasibility of expansion to 
collect data more state specific. This should be considered in 
conjunction with the further examination of appropriate state 
administrative records ..." 

adds to this: 

"There is no question that the state is the primary actor in 
education policy ... federal data collection should reflect this 
fact. Thus data should be collected on a district and state basis; 
if a sample of district data are collected ... the sample should be 
REPRESENTATIVE FOR EACH OF THE FIFTY STATES." 


The legitimacy of the NCES role in acquiring State data is claimed t 
to be based on the fact that NCES: 


"... operates in the non-political and professional tradii 
the U.S. Bureau of the Census and the U.S. Bureau of Laboi 
Statistics ... the NCES meets three criteria for an adequc 
system ... First, positioned at the federal level, it is i 
authorized to work with all of the nation's school distric 
states. Second, it receives an annual Congressional appr( 
of funds to be used for the express purpose of collecting 
education ... Finally and perhaps most important, it is it 
of the many stakeholders in education ... It is subject t< 
balance of power that affects all federal agencies." 

To IMPROVE the Federal-State administrative record data system, Leht 

that: 

"NCES should take the lead in developing a model state da 
and reporting system for district-level data. Although si 
may be collected and maintained at the state level, stand, 
public-use tapes from each state would be available." 

To which can be appended Natriello's proposal that: 

"A two stage process should be initiated, in the first s 
would identify several states interested in developing a 
data base relevant to state policy making. NCES would th 
with these SEA's to develop the data gathering procedures 
second stage NCES might select the most successful data b. 
and use it as the model for a national data base assemble 
collected by individual states." 


The natural evolution from those suggestions leads to a statement t 
would seem appropriate for NCES to stand ready to provide technical 
to states that request consultation on the best ways of collecting 
presenting their data" (W. Turnbull). 

Those are arguments for a state oriented collection schema, yet the 
concern expressed that: 

"... there are several compelling reasons to keep the bas 
collection at the federal level. First, for reasons spec 
earlier, most states and districts have historically coll 
minimum information about the elementary and secondary pu 
system ... Second, many public education issues require d 
comparable across states ... Or it may require the abilit 
differentiate general from state-specific problems" (Berr 
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redundancy and excessive data collection, audits and verification on 
local school data, and continued attention to equity results as well 
as excellence and achievement" (Cronin). 

ded to those concerns are those of McClure that: 

"While local, state and private sources can help to support the 
initiative, the federal government through NCES should set standards 
for data collection, insure comparability and timeliness, and 
provide computer networking systems for access." 

the current Federal-State basic data system, the "common core" administra- 
e data have an undetermined relationship to the data the individual states 
llect for their own management and oversight purposes. Nonetheless, Lehnen 
ccinctly states the key aspect of any Federal-state system must be a re- 
irement that, "the measures reported by the States and NCES must be the 
e." 


e current NCES education data acquisition program consists of two parts: 
tional sample surveys on topics and issues of national concern, and a 
sus, or 100% sample, of State and/or local administrative records collected 
om the State. The discussions of sample survey content, and the need for 
te-representative samples, are contained elsewhere in this document. In 
porting a further reliance on administrative data, the NGA suggests: 

"... that NCES do a comprehensive review of their data collections 
across subject areas to explore further efficiencies that could be 
realized through unduplicated data collection and more extensive use 
of administrative records. A single collection instrument that 
obtains relevant data for multiple purposes and users appears to be 
a far more efficient use of resources than multiple shorter surveys 
resulting in several sets of incompatible data." 

supporting the NCES program, the broadest such collection existant, the 
tional Education Association (NEA) states: "... the support and maintenance 
the Core component should be a national priority ... and the cornerstone of 
e educational information system in the united States.'* 


USING THE DATA, OR PROVIDING TEIE ROADMAPS FOR EDUCATION INFORMATION 

e two principal publications of the Center data are the Digest of Education 
atistics and The Condition of Education , both published annually. The 
gest has been published since 1962 as an abstract of statistical information 
education at all levels. It is a presentation of data tables from a 
riety of sources, both governmental and nongovernmental — and includes 
nimal text, and even less graphics. By presenting just tables, it leaves to 
e reader to read and interpret the numbers for him/herself. To increase its 
efulness, the Equal Employment Opportunity Commission has suggested that the 
gest should "provide interpretive analyses" to stimulate the reader. 


the supporting data table on the facing page. Buccino suggests aaaing 
part to the publication which "would comprise a collection of about fi 
substantial analytic and interpretive papers focusing on emerging issu 
review of status regarding continuing issues. (The plan for the 86 
Condition , scheduled to be released by June 1, 1986, calls for two sec 
indicators and essays.) 


Two suggestions for similar publications targeted at the state and loc 
level, where the majority of education policy and administrative decie 
take place, are offered by Hill: 

"An annual report on the status of education in each State e 
C ongressional district ... should not entail new data collec 
focus the reports directly on the members' own constituency 
deliver them directly and with some fanfare to the members' 
The design of such reports can be refined over time." 


and a series of optional, or on request, reports containing: 


"... information that LEA officials could use in reports to 
own school boards and the public. ... Because many district* 
the machinery and analytical talent necessary to use raw da 1 
NCES should offer participating school districts a menu of \ 
reports that could be created from the data being collected, 
reports could be simple tabulations and non-inferential sta 
that might be supplied with brief interpretive texts." 


Grant also suggests "... a definitive, comprehensive report on public 
elementary and secondary education ... prepared biennially for each s 

In framing State reports, Sims cautions us to note that "major policy 
decisions which affect education will not be made entirely in the SEA 
LEA's," and suggests: 

"The research opportunity presented to NCES, then, is to: asses 
information environment of the non-SEA education policymakers; r 
the results of this assessment; based upon this analysis, constr 
improved communication mechanisms to overcome the problems of le 
timing, relevance, personal predisposition, format, relationship 
constituent needs and peer thinking; test these mechanisms; and 
disseminate successful approaches to the State educational polio 
making community and those who serve them," 

Earlier this year NCES added the publication of Indicators of Educati 
Status and Trends, intended to present key variables in graphic or su 
table form, with a modest level of interpretive guidance to the gener 
reader. This seems to move somewhat toward the recommendation that: 

'... the Center should collect data and provide interpretat 
the data that are sufficient to give a "reading" of the gen 
health of the nation's educational system. Just as a physi 



education rather than attempt to collect extensive data on a large 
number of variables" (Peterson). 

Alternately that the Center should "... draw analogies in education to the 
statistics and indexes that are used in other fields, such as the gross 
national product, measures of housing starts, and the like" (D. Turnbull). 


TECHNOLOGY, DATA ACCESS, AND ARCHIVING 

Several authors offer various recommendations that NCEs move more towards 
electronic distribution and opening remote direct access to data bases. Sucl 
a move would provide an analyst with many more data than does conventional 
publication, and in a form/format which permits users to extract such 
informational content as is most relevant to their policy, research, or 
administrative needs. Among the suggestions are: 

"Accessibility is an absolute requirement of the system. ... 
downloading of files into other systems and highly sophisticated 
user friendly software so that questions can be addressed with 
minimal inconvenience. ... sponsor the development of expert systei 
to interface with central data bases. ... Accessibility could 
additionally include easy interface with graphic systems, 
statistical packages, and "what-if" scenario packages" (McClure). 

"... putting such a direct-access system in place — or as in the 
HS&B proposal, having it done by a contractor" (Coleman). 

"Establishing a central data library seems a natural role for the 
national education statistics agency. The availability of such a 
resource in-house might also have the beneficial side effects of 
keeping NCES staff in closer touch with developments in the states 
and providing means of cross-checking NCES' own data" (l3arro). 

"... a National Bureau of Educational Standards could serve as the 
central government repository and publisher of statistics on 
education in the U.S. ... as an archive of computer tapes of 
educational data that could be reproduced at cost by requests in 
writing, in person, or by telephone ... including ... data transfe: 
by computer" (Walberg). 


Walberg builds on the archival role for his proposed National Bureau of 
Educational Standards, adding that it; 

"... should be restricted to collection and assessment of data, 
calibrating and correlating measures, commissioning large-scale 
studies, making information available, and criticizing it. In thi 
way, it may provide good data for policy analysts and decision 
makers. It should, however, avoid political stances and 
recommending of policies and practices." 


Going Esther, Nalberg argues that thin la the time for nations statl 
an kinds to be coepnterlred. The Federal government is probably the 
sponsor that "could take on the large task of thinking through, oornmie 
and monitoring or conducting the research required to ^ l" * n 
place" that could set the standards for education statistics in the Ur 
states, including calibrating locally administered tests to a standarc 
order to provide comparison with other groups of students. His models 
this national Bureau of Education Standards are the National Bureau of 
Standards, the Library of Congress, and the Inter-university Consortii 
Political and Social Research at the University of Michigan. 


TECHNOLOGY AND DATA ACQUISITION 

There are many posibilities for using technology to increase the 
competitiveness and productivity of America's public schools. The gr< 
proliferation of microcomputers and personal computers in district of! 
schools, combined with growing sophistication about the use of such ec 
and telecommunications, are having increasingly far reaching implicat: 
future data collection. David suggests: 

"... now is the time to begin to design a computerized data 
collection system. Such a system will require considerable 
and testing; waiting until the technology is in place will ] 
a decade behind." 

She also suggest that: 

"The second application of technology that NCES should be 
investigating is the use of microcomputers for different ki. 
assessment instruments. The limits of paper-and-pencil tes 
well known. Designing new measures that go beyond simple m ; 
choice questions should be underway." 

Walberg amplifies a similar suggestion, stating: 

"If we start from the premise that we must inform citizens 
their schools; that educators should be informed about thei 
business including their costs, benefits, and views of citi 
that better education statistics may help us to understand 
our educational problems — then we need to think about har 
the vast powers of the computer, as other industries have d 
increase competitiveness and productivity. ... national hoo 
perhaps sponsored by the federal government, would make it 
to conduct sample surveys of districts, schools, and studen 
directly by computers. ... further advantage is the speed a 
surveys and tests can be completed. ... data are obtained m 
uniformly ... even analyses can be automated ... direct sam 
computer would make educational polls and national aesessme 
and cheap; they would minimize the total human time answeri 
questions yet provide more accurate estimates than far larg 
unsc ent f c surveys. ... Local, state or national asseesme 
spec al topics might be commissioned and comoleted in less 


general concept of longitudinal studies/surveys gains significant 
irsement from many of the authors, for example: 

"We endorse strongly the view that the longitudinal studies are a 
uniquely valuable educational resource and urge that they be 
designed as a long-term and recurrent element in the NCES 
data-gathering system" (W. Turnbull). 

"Longitudinal studies ... should be a priority to be maintained, 
improved in terms of data quality and potentially be expanded to 
gather more data, in terms of content, and sample size to make the 
data more state specific" (NGA). 

\ the substantial level of support of the concept of longitudinal studies, 
Lous authors have called for longitudinal studies covering different 
oents of the elementary/secondary grade span. Starting from the entry into 
educational system, proposals include: 

"... a longitudinal survey, with interviews of teachers and 
parents, to cover the transitions from pre-kindergarten to the early 
elementary school years" (Valdivieso). 

"... starting in 1980, a longitudinal survey of second graders and 
their parents in a sample of elementary schools that feed into the 
high schools selected for NELS-88" (Bishop). 

"... students in the elementary, middle-school, and junior high 
school grades. ... important ... since a number of problems 
associated with high school students (e.g., teenage pregnancy, 
dropping out, drug usage) are now seen to have their roots in the 
years prior to high school ... impact of family on early schooling 
... impact of various school-to-school transitions on young students 
... the effects of classroom organizational characteristics ... and 
the effects of the instructional and evaluative strategies adopted 
by teachers" (Rosenholtz). 

"... from the middle school years through high school and beyond ... 
our ability to analyze and understand other high school processes is 
also limited by studies which gather initial data on 10th graders. 
The phenomena of tracking and ability grouping is well underway 
prior to 10th grade in almost all high schools — the determinants 
of these assignment practices are operating by 7th and 8th grade. 

... inferences about the effects of high schools on students are 
necessarily limited if analysts do not have data on students that 
preceeds the entrance of the students into high school" (smith). 



"... a national longitudinal effort at comparing public and 
private school sites across regions would provide researchei 
policymakers with integrated, reliable information ... about 
relationship between school site investments and educational 
outcomes ... to track the progress of school reform initiate 
(McClure). 


EDUCATION RATHER THAN SCHOOLING 

Finally, "NCES should explicitly consider moving beyond the collectioi 
on schooling to the collection of data on education 1 ' (Natriello). 

This might mean: 

"Most current NCES data collection activities focus on elem< 
and secondary schools. While schooling should remain at tin 
NCES data collection plans, ... students are being exposed ' 
growing number of learning resources outside of the traditi* 
schools. ... Non-school educational activities may become ii 
control variables, much like parent educational levels and ■ 
resources, in understanding the effects of schooling." (Nat: 

"The federal statistics program seems to define public schot 
K-12 terms. Yet schools everywhere are looking at pre-schoi 
after-school care, and many forme of adult education and se: 
Whom to serve, and how to render services to new population) 
policy issues at local and state levels which NCES may be al 
illuminate with trend data." (National School Boards Associ, 

"... education data in the future should not only be derive 1 
schools and other formal educational institutions, but also 
other deliverers of educational or training services such ai 
private sector, the military, voluntary associations and th 
countless other organizations and agencies which provide ed 
and training services. Education must be defined more gene 
to consist of more than just schools." (Usdan). 

To aid such a move to consider the youth population as a totality, ra 
focusing solely on those in conventional K-12 or 1-12 schools, the NG. 
proposes: 


"While education systems do vary widely across states; it a 
that states would welcome common reporting on certain natio 
elements that would allow valid comparisons to be made. ... 
as the major statistical agency responsible for labor force 
statistics has defined the population (16 years and older) 
mutually exclusive categories ... NCES as the major statis 
agency responsible for education statistics, should conside 
defining the population (0-16 years old) in a similar fashi 
mutually exclusive categories. This would help in the deve 
of definitions," 


Authors say that NCES program staples — mail sample surveys and 
administrative record censuses — are not the only appropriate acquisition 
codes for needed education data. Certain research issues may call for 
different designs beyond the large survey or universe data programs involving 
paper forms or electronic transmission, and self-response: y 

«... laboratory experiments or observational studies are probably 
preferred to survey techniques for studying the processes" 
(Berryman). 

Additionally they may call for: 

"... in-depth case studies such that each study can help illuminate 
the meaning of organizational efficacy for a particular school and 
help us generalize to that level of a critical mass of attributes 
needed under different conditions to achieve specific purposes for 
any school" (Hersh). 

NEA suggests that NCES: 

"Consider expanding the NCES program to include case studies, field 
studies, policy reviews, historical research, and additional surveys 
to expand the scope and detail of effective schools data." 

Hawley refers to the pitfalls in and limits of statistical survey data, 
stating: 

"Analysis of large scale statistical data should lead to and be 
informed by more intensive research that examines the dynamics of 
student and teacher interaction and otherwise helps us to avoid 
false assumptions about the meaning of statistical data. ... one 
cannot determine how an evaluation system ... is implemented and how 
context affects implementation without using research techniques 
that are more intensive than statistical surveys" (Hawley). 

Selden, in what would be a dramatic departure from current practice, wants 
national sample surveys to provide standardized teacher performance data based 
on classroom observations: 

"With definitions and operating procedures (which some states and 
many local school districts are developing in order to evaluate 
teachers) a national ... sample of teachers would be observed 
periodically to provide longitudinal and comparative data on the 
overall pedagogical ability of teachers." 

Many of the suggestions contained in the set of papers will require investi¬ 
gative research to define the concepts necessary before metrics can 


be specified and developed. Several have suggested that NIE and NCES * 
cooperate wherever metrics are to be developed. It has been further si 
that: 

"NCEs should work closely with the appropriate National InstJ 
of Education Research and Development Centers and Regional Ec 
tional Laboratories to coordinate nationwide surveys with th< 
on-going work of these major government sponsored, educations 
research institutions" (Natriello). 

Peterson, making the same suggestion, notes that: 

"One productive avenue for collaboration would be for NCES t< 
closely with several of the ... soon to be funded ... Centers 
has requested that each of these Centers reserve 10% of thei: 
budgets in 1987 through 1990 for collaboration with other nai 
Centers." 


8EY0ND THE DEPARTMENT OF EDUCATION 

With respect to other agencies' data, Bishop makes specific recoounendai 
for improvements in the October education supplement to the Current Poj 
Survey, and also the 1990 Decennial Census of Population and Housing, \ 
which suggest greater coordination with the Bureau of the Census staff 
concerned with education matters. Other inter-governmental suggestions- 
include: 


'Finally, in the next decade, NCES might improve the relevant 
technical quality, and utility of their data as well as imprc 
cost effectiveness of their data collection efforts by collat 
with other large organizations, both in planning data collecl 
in gathering the data" (Peterson). 


"Consider establishing a systematic (data gathering and anal) 
research program coordinated with Health, Labor, the Census, 
HUD ... First, I am sure that there are inter-governmental 
coordination committees that meet every two months to share 
information to help all interested parties in the government 
abreast of the latest and most effective methods of data gatl 
and analysis ... Second, X am confident that there are a subs 
num er of ideas in the field about ways of improving data gal 
and analysis strategies that could use some stimulation and 
direction and could provide great savings to the federal govs 
in the very near future" (Smith). 


One way of beginning to ameliorate these problems (of inte 

definitional a nd quality comparability) won] 
have “CBS assume responsibility for coordinating U.S. invol 
XEA activities" (Smith). 
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-fe provide suggestions on the following topics: 


O) Consistency of Definitions 

( 2 ) Cross-Sectional and Longitudinal Studies 

(3) Process Data Collection 

(4) Competency-Based Curriculum 

(5) Ways of Improving the Quality of the Data Collection Process 


Consistency of Definitions 

The most clearly stated data collection need addressed by the writers of the 
papers synthesized in this report is the need for clear, consistent definition 
Without consistency, the reliability and validity of the data collected from 
various sources become questionable and the value of the data is greatly 
reduced. 

Soae papers either explicitly or implicitly suggest that the best way to obtai 
consistency is for the federal government to mandate standard definitions or f 
the states to agree upon and use a standard set of definitions. Others sugges 
that the fedoral government use strong-arm tactics in pressuring states to use 
standard definitions. 

Local autonomy is sacred in many state educational systems. Some states are n 
politically in a position to mandate standard definitions. This is especially 
true in areas related to the curriculum. Standard definitions may be suggeste 
but it would not be feasible, for example, to require all systems to adhere tc 
the same nomenclature in providing titles for courses. 

States should be encouraged to agree upon a recommended set of standard 
definitions and should be encouraged, but not coerced, to use these definitior 

NCES should develop incentive programs that encourage the consistent use of 
standard definitions. Incentives could include computer software packages th« 
ate useful to school systems in the processing of data, free reports, user 
tapes, free computer time for accessing information files, technical services 
and other benefits suggested by local and state school personnel. 

Incentives coupled with an awareness program designed to develop an 
understanding about the need for consistent definitions may prove to be the mi 
effective way of obtaining reliable and valid data. 


Awareness, incentives, and quality control techniques that identify questionab 
data are the best ways to control data collection problems asaociated with the 
"consistency of definitions." 


In a later section of these comments, we address the training needs to improve 
consistency of definitions in federal and State data systems. 

Cross-Sectional and Longitudinal Studies . 

An over-interpretation of the strong support for longitudinal studies containe 
in the synthesized papers could lead some individuals to conclude that 
cross-sectional studies are not appropriate for generating NCES time series 
data. Very few papers addressed cross-sectional designs. 

Cross-sectional designs can provide useful time series data. The data are oft 
less expensive to collect than longitudinal data, because individual students 
not need to be traced. The data can be ueed to establish trend information an 
to corroborate information collected from smaller longitudinal samples. 


One caution is highlighted for emphasis. National Assessment does some things 
extremely well in measuring student performance, much better than the 
longitudinal studies which have relied on paper and pencil multiple choice typ 
items to measure student performance. Historically, National Assessment has n 
relied entirely on the multiple choice question for the collection of 
performance data and much additional information has been collected, for 
example, in writing and mathematical problem solving. 


If National Assessment and the longitudinal studies are combined, the best of 
both should be used. This would result in a massive and expensive study, but 
one whose value would exceed the value of the combined results of two or more 
individual studies. But, the expense may prove to be the fatal flaw in the 
combination strategy. As ways to reduce the expense are considered, the 
strengths of the individual systems may be lost. 


If saving dollars is the motivating factor for combining cross-sectional {NAEP 
and longitudinal studies, care should be exercised. However, if the more 
effective use of dollars is the motivating factor, serious consideration shoul 
be given to combining the National Assessment and the longitudinal studies. 


Before NCES reaches a decision on this important and complex design issue, we 
recommend that they commission a collection of papers where the pro's and con 
would be identified and considered on the appropriateness of combining the 
National Assessment and the longitudinal studies, e.g., the advantages and 
disadvantages of alternative sampling and analytical design options, appropri 
ways to measure student performance relative to the response burden imposed c 
e s u en an the school, and cost effectiveness. Such papers could serve 
* p? S f 1 ® ul f te P ublic debate and subsequent input into NCES before they 

, a . na 6G slon as to design that is of the most benefit and least 
cost for NAEP or NELS or their combination. 


several papers focused on the need for collecting process data. 0 ft*n t 
jata can be collected by compiling information contained in administtati. 
records or though the use of survey questionnaires, but some process dut » 
require the use of observational or interview data collection technique j. 
data collection techniques are expensive to implement. The resulting . 
say bo to delete process data collection activities. 

1ICES should consider the funding of multiple, small process-oriented at;., 
studies using the same research design, conducted by individuals who huv-.- . 
data collection costs and low overheads. Macro analysis of the nultLple 
studies would provide information comparable to that expected in a large 
study and at a reduced cost. 

Coppetency-Based Curricula 

Ihere is some evidence to suggest that there is a movement toward 
competency-based curricula. This movement may have a dramatic impact on * 
measurement of student performance. 

A9 school systems gain a hotter understanding of their goals and their 
objectives, they may become more interested in making sure that performing 
r.easures relate to instructional programs. School personnel may expect t.. 
teats to measure what is being taught and not what is minimal in coverage i 
easy to measure. 

NCES should make sure that performance measures reflect the objectives or 
coapetency-based curricula. Not all objectives could be included, but rte.i . 
that are used should be drawn from explicitly stated instructional object;, 
coioon to an identifiable set of curricula. 

One additional caution is provided. Those who depend on latent trait the u 
equating tost items, which assumes that learning is an hierarchical proce^. 
should review the technical and political discussions and the resulting e:..i 
iw assossmont methodology that took place in Great Britain, 

loproving the Quality of the Data Collection Process 

In our judgment, we can achieve a higher quality of data systems at the 
national, State, and local level, by improving the processes that are use; 
design, collect, process, analyze, and report educational data. We provi :? 
following suggestions: 

11) Provide comprehensive guidelines and staff training in the preparation 
Request for Proposals (RFP's) for data collection contracts. Items a:, 
issues that should be covered are: 

(a) A description of the intended respondent population. 

<b) Any limitations on response burden should be specified. 

lc) Timelines of the schedule for the project should be realietic 

for the work to be accomplished. 


data are the best ways to control data collection ptouxemo 
"consistency of definitions." 

In a later eection of these comments, we address the training needs to improve 
consistency of definitions in federal and State data systems, 

Cross-Sectional and Longitudinal Studies . 

An over-interpretation of the strong support for longitudinal studies contained 
in the synthesized papers could lead some individuals to conclude that 
cross-sectional studies are not appropriate for generating NCES time series 
data. Very few papers addressed cross-sectional designs. 

Cross-sectional designs can provide useful time series data. The data are of ter 
less expensive to collect than longitudinal data, because individual students dc 
not need to be traced. The data can be used to establish trend information and 
to corroborate information collected from smaller longitudinal samples. 

One caution is highlighted for emphasis. National Assessment does some things 
extremely well in measuring student performance, much better than the 
longitudinal studies which have relied on paper and pencil multiple choice type 
items to measure student performance. Historically, National Assessment has noi 
relied entirely on the multiple choice question for the collection of 
performance data and much additional information has been collected, for 
example, in writing and mathematical problem solving. 

If National Assessment and the longitudinal studies are combined, the best of 
both should be used. This would result in a massive and expensive study, but 
one whose value would exceed the value of the combined results of two or more 
individual studies. But, the expense may prove to be the fatal flaw in the 
combination strategy. As ways to reduce the expense are considered, the 
strengths of the individual systems may be lost. 

If saving dollars is the motivating factor for combining cross-sectional (NAEP} 
and longitudinal studies, care should be exercised. However, if the more 
effective use of dollars is the motivating factor, serious consideration should 
be given to combining the National Assessment and the longitudinal studies. 

Before NCES reaches a decision on this important and complex design issue, we 
recommend that they commission a collection of papers where the pro's and con's 
would be identified and considered on the appropriateness of combining the 
National Assessment and the longitudinal studies, e.g., tho advantages and 
disadvantages of alternative sampling and analytical design options, approprlat 
ways to measure student performance relative to the response burden imposed on 
the student and the school, and cost effectiveness. Such papers could serve as 
a basis to stimulate public debate and subsequent input into NCES before they 
reach a final decision as to the design that is of the most benefit and least 
cost for NAEP or NELS or their combination. 



Proce ss Data Collection 

Several papers focused on the need for collecting process data. Often th 
data can be collected by compiling information contained in administrate 
records or though the use of survey questionnaires, but some process data 
require the use of observational or interview data collection techniques, 
data collection techniques are expensive to implement. The resulting dec. 
may bo to delete process data collection activities. 

NCES should consider the funding of multiple, email process-oriented stud 
studies using the same research design, conducted by individuals who have 
data collection costs and low overheads. Macro analysis of the multiple i 
studies would provide information comparable to that expected in a large i 
study and at a reduced cost. 

Competency-Based Curricula 

There is some evidence to suggest that there is a movement toward 
competency-based curricula. This movement may have a dramatic impact on 
measurement of student performance. 

As school systems gain a better understanding of ttieir goals and their 
objectives, they may become more interested in making sure that perforraani 
measures relate to instructional programs. School personnel may expect ti 
tests to measure what is being taught and not what is minimal in coverage 
easy to measure. 

NCES should make sure that performance measures reflect the objectives of 
competency-based curricula. Not all objectives could be included, but me. 
that are used should be drawn from explicitly stated instructional object 
common to an identifiable set of curricula. 

One additional caution is provided. Those who depend on latent trait the 
equating test items, which assumes that learning is an hierarchical proce 
should review the technical and political discussions and the resulting c 
in assessment methodology that took place in Great Britain. 

Improving the Quality of the Data Collection Process 

In our judgment, we can achieve a higher quality of data systems at the 
national, State, and local level, by improving the processes that are uee 
design, collect, process, analyze, and report educational data. We provi 
following suggestions: 

(1) Provide comprehensive guidelines and staff training in the preparati 
Request for Proposals (RFP's) for data collection contracts. Items 
issues that should be covered are: 

(a) A description of the intended respondent population. 

<b) Any limitations on response burden should be specified. 

(c) Timelines of the schedule for the project should be realistic 
for the work to be accomplished. 


(d) Consistency within the Ktr ie - 

mission of the RFP review dates, deliverables, major project 
activities and responsibilities should be checked Cor consis 
tency throughout the work statement of the Rl-P. 

(e) The research questions and policy issues that need to be 
answered should be stated, 

(f) Levels of effort should be clearly stated. 

(g) The statistical precision requirements should be 3tated and must 
be reasonable for the level of effort specified. 

(2) Provide a forum to address the politics of data collection for NCUS and 
other Federal government data collection officials to discuss the 
variations that exist between individual states, e.g., ouch as those states 
that have a highly centralized administrative structure versus thoso that 
are dscentralized. 

(3) Develop and provide training and technical assistance to NCL.S and State 
Department of Education staff on the fundamentals of data collection 
practices and how to develop standard operating practices (SOP) for data 
collection. Such services should include the following: 

(a) Training in the definitions of the variables of the data to be 
collected, including an understanding of the practical aspects of 
the concept of data accuracy (validity, reliability, 

sampling, and completeness). 

(b) Training in the fundamentals of data handling and processing practices 
including the practical "How To's" of the development of in-range and 
consistency data edits, database creation and organizational 
techniques, advantages and disadvantages of alternative aggregation 
levels of the data, and appropriate data linkage techniques. 

(c) Timeliness - the issues and importance of reporting data when it 
can be useful to policy makers. 

(4) Encourage postsecondary educational institutions to include training in the 
fundamentals of survey research in their curriculum for future educational 
researchers, e.g., the practical methods of sampling, including an 
understanding of both sampling and NON-sampling errors, practical survey 
design and data collection methods, practical aspects of data handling and 
processing, and experience in analyzing actual data sets and report 
writing. 


The collection and reporting of data has historically been among the prime 
missions of the U.S. Department of Education. In fact, if we go back to the 
original post-civil War charter, data collection was the only mission. 

In the intervening decades, especially since the enactment of major new prog 
in the mid-1960's, data collection has moved more and more into the backgrou 
of the Department's activities. In fact, today the resources dedicated to t 
area are infinitesimal — far less that one-tenth of one percent of the tota 
budget of the Department of Education. 

Ironically, in 1985 the American public and its political leaders have devel 
a voracious appetite for data on what we are getting for our investment in 
education. 

In my view, one of the major reasons for the absence of good outcome data is 
that for the past two decades the data collection efforts of the Department 
been driven by issues related to input, processes, and finance. important a 
they are, they cannot be allowed to drive out what must be our overriding 
concern for outcomes on what students know and how can they apply that 
knowledge. 

There are also a number of other issues of concern to me, including: 
the use of new technologies, 
the need to acknowledge non-school learning, 
the potential for reorganizing NCES, and 

the need to have a system that is responsive to current, not merely 
historical , concerns. 


In preparing the chapter on educational outcomes, I was struck with how much 
writers responded in accordance with the system, the tests, the measures tha 
know today. None of the writers said, "Let's step back and see what new can 
done." Few even ventured into new technologies. In my view, a major challe 
to NCES will be the design of a data collection system that both looks to ft 
issues and utilizes new technology to the highest degree possible. 

I am also concerned that too much emphasis seems to be placed upon the exist 
educational structures, e.g., secondary schools, elementary schools, and not 
enough attention is given to the impact of non-school factors in educating 
people in our society--o.g., television, newspapers, museums, magazines, 
corporate training, military training, and so on. It is the impression of t 
observer that much learning does go on in these settings and that it needs t 
both acknowledged and measured. 



the collection of information relevant to our formal school structure or as ti 
collection of information on the knowledge (education) of our citizens. In tl 
final analysis, do we seek data on schooling and credentials (diplomas, degree 
etc. ) alone or on the overall skills and knowledge of our people? 

The above is not meant to eliminate the critical focus on measuring what our 
schools are accomplishing. To the contrary, I would agree that it is most 
difficult to judge the impact of schooling unless we have a baseline of 
information relative to non-school impacts. What is the value added by our 
schools? 


To make an extreme case, does a young child know how to read because of Sesame 
Street, parental tutoring, or the local school? Or, is it because of all thre 
At the other end of the scale, does a 17-year old know how to make change and 
compute sales tax because of school or on-the-job training or both? 

An argument can also be made that it is a mistake to adhere too closely to the 
line that separates secondary education from post-secondary. For example, is 
there a real distinction between learning in certain high school programs and 
community colleges? How many colleges pour resources into remedial programs? 
The point hero is that we need to address issues of articulation between 
elementary and secondary, and between secondary and post-secondary. Simply 
because of the way NCES is organized, we should not lose these issues. A 
radical suggestion: Perhaps NCES should consider a reorganization along line* 
that would cluster together resource issues in one division, output issues in 
another, and process issues in a third so that there can be continuity in thei 
areas. Under this proposal, the distinction between the elementary and 
secondary division on the one hand and the post-secondary division on the oth< 
would be eliminated. 

A final thought: Although it is important that NCES efforts emphasize the 
long-term collection of significant data on important non-trendy issues, it ii 
perhaps equally important that NCES also have the ability to collect informat, 
on issues of current importance, e.g., school violence. Although only one paj 
mentioned this issue, it has become perhaps the most important political issuf 
in education. To my knowledge, the last real data collection effort of any 
magnitude on this subject was done nearly 10 years ago. Yet, there is abundai 
evidence that this is one of the most important issues to parents, to 
communities, and to many, many students. NCES should not ignore the issue, 
and it cannot be addressed simply through a fast response survey. We need, a 
the very least, trend data and incident data. 

The NCES redesign project is both ambitious and encouraging. Soliciting inpu 
from large numbers of individuals and associations has been a courageous, eve: 
heroic, undertaking. I am very pleased to have been asked to play a small pa 
in the effort. 


statement by MARGARET K. GWALTNEY 


The National Center for Education Statistics invited more than 50 educational 
researchers and practitioners and other users of educational data to provide 
recommendations that would assist NCES in the redesign of its elementary and 
secondary education data program. The Center ashed individuals and education 
associations to identify specific data elements that would "provide the 
necessary information in support of present and future government, business, an 
academic decisionmaking, and that can help inform the American public." 

The response to NCES's invitation for comments was encouraging. The papers wer 
provocative and full of ideas and recommendations that, if implemented, would 
surely enhance the current NCEs data systems. The writers were unrestrained by 
budgetary concerns or, in some cases, even political feasibility. The result 
was a wide range of recommendations, both for the collection of new data and th 
conduct of different types of activities, such as research and analysis. 

It is time now to share the recommendations made by writers with other writers 
and the education community and public at large, to get others' reactions to th 
recommendations that were made, and to sift through the recommendations and com 
up with an agenda, a plan, for NCES's next decade. Among the most important 
considerations will be the appropriate mission for NCES. The question of 
whether the Center should become involved in activities other than data 
collection and reporting must be decided; the calls for further involvement in 
research and analysis must be answered. 

NCES Mission 


Ihe NCES mission is currently not well defined. Some writers point this out in 
their papers. Now is an appropriate time to clarify the mission and to broaden 
it if that is deemed worthwhile. 


Given resource constraints within the federal government and within NCES, the 
mission of the Center may not be able to be broadened to the extent that many 
have recommended. Indeed, it may not even be appropriate to do so. We must 
remember that the purpose of a national data system is to collect data that wil 
inform the public and educators about the quality of education and other 
national trends—e.g. , the shortage of mathematics and science teachers, the 
drop-out rate, and so on—and that will assist state and local educators in 
their design and implementation of an effective education system. NCES, 
however, must not be expected to provide all the data these educators may need, 
b ut only those data that reflect issues of national concern and importance. 
States and local districts must naturally still be involved in data collection 
relevant to local issues and problems. 


NCES also should not be in the business of program evaluation, as some have 
suggested, „ or should it shift its emphasis to research and analysis, althougl 
some special studies on topics of national importance may be approprtate. 


assess school effectiveness and tne national conuxtxon ox euuuduuu. xuusiuej 
able research is needed, but this responsibility should be assumed by other 
federal agencies and private research institutions. 

Obviously, the research questions should not and cannot be ignored. In fact, 
NCES should work cooperatively with such organizations as the National 
Institute of Education, the Department of Labor (Bureau of Labor Statistics), 
the Census Bureau, and the educational laboratories and centers, among others 
to ensure that the research it needs is conducted. This research should focus 
on identifying those variables having to do with school effectiveness, which 
once identified should than be added to its data collection program. The 
mission of each of these other agencies with respect to educational data col¬ 
lection should be delineated and made explicit in the agencies' mission state¬ 
ments. This type of cooperative planning effort, not just with non-federal 
groups and individuals but also with other federal agencies such as NIE, is 
essential. Not only will such an effort yield a better and more complete NCE! 
data program, but it will also result in a better educational information bas« 
mors generally. 

Information Dissemination and Utilization 


Pew writers addressed the issues of dissemination and utilization. However, 
implied by many of the comments made in the papers was that the data that are 
available are not always presented in a way that is most useful to potential 
data users. The data are not always timely, nor are the data always displaye< 
in a format that makes that data moot accessible. NCES should therefore exam, 
new ways to make the data it collects moro useful to educational policymakers 
and practitioners and educational researchers. In particular, it should 
investigats how it can expand its user group. The needs of state and local 
educators should be of particular concern. However, NCES must remain cautiouj 
about how far it goes to meet local and state needs. As stated previously, N( 
must avoid becoming involved in data collection to serve specific local and 
state needs. The purpose of the NCES program should be to maintain a nationa. 
educational data base that addresses educational issues of national concern. 

To ensure continued and increased interest in the data it collects, NCES woul« 
also be wise to seek advice and recommendations from its user community 
periodically, in a manner similar to the way it has sought input during the 
redesign process, but on a smaller scale. The recent process has been extremi 
productive. And, sven though NCES will not be able to implement all the 
recommendations made by those who were invited to write papers, it has captur 
the attention of a large number of users during the process. These individual 
associations, and others will surely be watching NCES over the next months an< 
years to see how it enhances its data program. They are likely to become mori 
aware of and perhaps even greater users of NCES data as a result of their 
participation in the redesign process. To increase the number of users over 
the next decade, NCES should continue to seek input from the potential user 
community, particularly from those types of users who in the past have made 
limited use of the NCES data. 


Statement by WARD S. MASON 


In preparing the chapter on process variables I tried to reflect the ideas of 
paper writers without injecting my own perspective. I welcome the opportunity t< 
offer a few personal observations based on study of this valuable collection of 
papers. However, I will not limit myself to process variable issues. 

Narrowing the Purpose of a National Data System , A dominant impression from 
this experience is that, collectively, the recommendations are overwhelming in 
variety and detail. This was expected, since no particular restraints were 
placed on the writers. But as a result, the problem of refining our concept of 
what is appropriate for a national data system and what should be the mission oi 
MCES has achieved renewed saliency. The two dimensions of this problem are: 

o What kinds of data are appropriate to a national data 
system as opposed to that for State or local systems? 

o What kinds of data are appropriate for a national system 
of education indicators designed to measure the 
"condition and progress of education "as opposed to that 
appropriate for special research studies, program 
evaluation, diagnostic instruction, or other special 
purposes? 

The first issue becomes "sticky" because of our federal system. We want 
national data because we need to know as a nation what the quality of our 
educational system is, but we do not have a national system of education. The 
data system must be designed to bo useful to State and local policy makers as 
much or more than national leaders. National aggregates require comparable 
data. But to the extent that our State education systems differ, they may 
require different data systems to monitor their different structures and 
different policies. The national system should confine itself to the major 
structural components of the educational system that are common to ell states. 
State data systems can add detail to suit their particular educational 
philosophies and policies. However, as suggested by one writer (Natriello), 

NCES might well take on the task of assisting States in the development of a 
pilot state data system. 

The second issue arises because, in ray view, many of the suggestions made by 
paper writers are interesting questions for research or program evaluation, but 
are inappropriate for a data system. The research issue has two dimensions. 
First we need a research base which establishes the role and importance of 
particular variables before they are incorporated in a data system, and in many 
cases that research base does not exist. Further, the development of appropriat 
measures is prerequisite. So some variables are likely candidates for a data 
system, but can only be incorporated after the necessary research and 
development have been accomplished. 

Second, some questions are not amenable to answer from a data system but requir 
special research studies. Questions such as the efficacy of particular 


factors * Major longitudinal studies such as High School and Beyond can go into 
lore detailed issues than the Common Core, but here too there are limits. For 
ixample, such a study might be able to obtain a simple characterization of an 
.nstructional procedure such as "used team teaching", but would be unable to 
>btain detail on what division of labor, the characteristics and qualifications 
if each team member, etc. 

f this position is accepted, then it would be appropriate to view many of the 
mggestiona as ideas for a research, development, or dissemination agenda 
ippropriate to the National Institute of Education (or its successor units) as 
fell as to the research community generally. I think that NIE should be brougl 
.nto the redesign process and explore how it might pick up on some of these 
Ideas, either by itself or in cooperation with NCES, 

; am also concerned with the danger implicit in many of the suggestions for wh< 
ire program evaluation functions. Many writers believe that a national data 
iystern should be able to tell us "what works". This seems like a natural 
ixpectation, but one which 1 believe is doomed to frustration. In the social 
ind behavioral sciences, the problems of multiple causation are so complex thai 
>ur ability to sort out the causes of particular results is very limited. Sue) 
knowledge as we can obtain must usually couie from detailed field studies, and 
iven there it tends to be time and context specific. 

Je would do better to confine our attention to establishing good measures, 
:ogether with absolute and comparative standards, of "how we're doing", and 
.eave program evaluation to special studies. 1 believe we can establish such 
leasures and standards for all kinds of variables (i.e. input, process, and 
mtcorae varables). Indicators of effective schools or effective teaching are 
sxamples of measures of the quality of schooling and the quality of teaching, 
respectively. They are based on research about what factors are important in 
.raproving student outcomes, but the data system can be used to validate the 
research in only the grossest manner. We may wish to determine whether reform 
.s working in those states that have adopted it. We can assemble the outcome 
.nforraation for those states, but even given dozens of variables and controls, 
fill we be able to trace the effects of the reform to the outcomes, ruling out 
ilternative explanations. It seems doubtful. 

'roblems of Utilization . Little attention was given to the problems of data 
itilization. There were some suggestions made concerning the need to provide 
locuraentation on definitions, how the data were collected, etc. and to prepare 
tnalyses and interpretations that might reduce the number of misinterpretation 
if findings. But relative to use by policy makers, there seems to be an 
inderlying assumption that the date will be used for rational problsm-solving 
n instrumental way. Yet there is a considerable body of research on utlizati 
»f knowledge which shows that that is not usually the way it happens (Charles 
lindblom and David K. Cohen, Usable Knowledge: Social Science and Social 
roblem Solving . Yale University Press, New Haven, 1979), More often, data 
erve an "enlightenment function". Weiss sums up the argument in this way: 

"Research does not solve problems; it provides evidence 

that can be used by men and women of judgment in their 


efforts to reach solutions. It helps to establish the 
premises on which the debate shall take place, providing 
an orientation, a language of discourse, and a conceptual 
base for the discussion of policy" (Carol U. Weiss, 
"Improving the Linkage between Social Research and Public 
Policy", in Laurence E. Lynn, Jr., (Ed.), Knowledge and 
Public Policy: The Uncertain Connection , National Academy 
of Sciences, Washington, D.C., 1978, pp. 76-77.) 


•iitchell has pointed out that policy making is a process, not an event, and go 
hrough several stages. Data and knowledge may play different roles at each 
stage of the process. Thus, in Mitchell's formulation, in the initial 
articulation stage of defining the problem, the conceptual frameworks of 
researchers may be important in providing new perspectives on where the proble 
ios; at the aggregation stage, problem solving is appropriate; the allocation 
stage requires evidence assessment; and during oversight, performance evaluati 
comes into play (Douglas E. Mitchell , "Social Science Utilization in State 
Legislatures", in David C. Berliner (Ed.), Review of Research in Education , Vc 
9, American Educational Research Association, Washington, D.C., 1981, pp. 
257-308). 

Models of Schooling . I agree with the point made by a number of writers that 
ata systems need to be based on, theoretical models. It is noteworthy that me 
of those suggested were essentially economic models; input/output or 
productivity models. 

American schools have been characterized in the past as having been based on c 
factory model, and one can see how many of its characteristics prepare otudenl 
for work in the factories (or offices?) of the industrial age. Peter Drucker, 
attong others, has made the point that, as we move into poet-industrial society 
the economic firm of the future will be information based. (Peter Drucker, WaJ 
Street Journal , Jan. 9, 1985). If that is so, I think it would be appropriate 
to start developing the information-based school. Walberg's suggestions 
regarding the use of adaptive testing are among those compatible with this 
thought. 

ro move in this direction it will be necessary to change some very basic 
attitudes that lead people to to see information collection as a burden. Sur 
it is a burden when it is not functional. But when information comes to be o 
direct use in diagnosis and instruction, those attitudes can change. At scho 
sad district levels, management information systems can grow more functional. 
ICES, perhaps with NIE, might very well assist in the development of informat 
systems for instruction and management. This would help to insure that at 
Joints of common Interest there would be common data elements and the ability 
ise data from school data systems as input to national data systems. However 
the central point is that if schools begin to model the information-based 
organization, then they will, through a new "implicit curriculum' 1 , help prepa 
students for the world of the future. 

Students at Risk . Picking up on an idea presented by Valdivieso, I would lik< 
to see a construct of "students at risk" developed. Too often we use 


there is tremendous variation witnm most aemograpnrc categories ana tnereror* 
the relationship between the category and disadvantaging condition is likely 
be tenuous. It is time to put aside the proxy variables and study students ii 
terms of the specific conditions that place them at risk. These conditions ai 
their relationship to instructional strategies and performance need to be 
studied singly. It might also be possible to combine some into indexes. 

Teaching Careers or Education Careers . The implicit assumption of many paper* 
dealing with problems of attracting and keeping a quality teaching force is tl 
teaching is a career unto itself and that our goal should be to retain good 
teachers in that role. Yet it is also traditional in education to expect tha 
administrators, curriculum specialists, and other professional staff will be 
drawn from the teaching ranks. I think it would be immensely helpful if we 
could broaden our horizons to include all of the education profession and 
visualize a variety of career paths, some of which would lead out of teaching 
and some of which might combine teaching with other responsibilities. This 
perspective would have an impact on studying the teacher supply and demand 
problem. 

External School Improvement Resources . In recent years an extensive 
infrastructure has emerged consisting of organizations designed to provide 
improvement assistance to schools in the form of applied research, informatioi 
training, and technical assistance. Some of these capacities are found in thi 
central offices of medium to large school districts, while others are found 
outside local school systems. They include educational information centers, 
curriculum materials centers, teachers centers, technical assistance centers, 
etc. operated by state and intermediate education agencies, bureaus of field 
studies in colleges and universities, regional educational laboratories, 
research and development centers, and profit and not-for-profit organizations 
the private sector. Knowledge of these resources is a gap in the present dat< 
system. NIE compiled a directory of these organizations in the late 70's*, ai 
some of the regional laboratories have attempted to keep this kind of 
information current on a regional basis. Information on organizational purpoi 
types of services, types of staff, and source and level of funding provide roi 
measures of the assistance capacities available. Data of this nature collecti 
on a state basis would provide further clues on the capacities of each state 
effect school improvement. 

State Policies . The writers showed a great interest in state-to-state 
comparisons of input, process, and outcome data. They also wanted to use the 
data to monitor the impact of particular reforms. While I have expressed 
skepticism about the use of data systems for making such evaluations with any 
precision (see above), a knowledge of the laws, regulations, and policies of 
each state would seem to be needed for the interpretation of state comparison* 
The Education Commission of the States has provided compilations of this type 
from time to time. However, more work is needed on developing a typology of 
state policies. Collection of such data on a biannual basis would seem 
appropriate. 


*Rolf Lehming (Ed.), Directory of Research Organizations in Education: Reeear* 
Development, Dissemination, Evaluation, and Policy Studies. Par West Laboratc 
for Educational Research and Development, San Francisco, CA 1982. 




appendixes 





The Papers, Comments, and Letters 
Incorporated into this Public Discussion Draft Synthesis 


James M. Banner, Jr. - Council for Basic Education 
"Revising Educational Statistics" 

Stephen M. Barro - SMB Economic Research, Inc. 

"NCES Data on School Finance and Teachers: 

Assessment and Recommendations" 

Sue E. Oerryman - The Rand Corporation 

"Education and Employment: Substitution Possibilities and 
The Teacher Labor Force: Supply and Demand" 

John H. Bishop - The National Center for Research in Vocational Education 
"Data Collection for Improving Eleraentary/Secondary Education" 

Alphonse Buccino - The University of Georgia 
"Monitoring the Condition of Education" 

James S. Coleman - The University of Chicago 

"Data Needs for School Policy in the Next Decade" 

Joseph M. Cronin - Massachusetts Higher Education Assistance Corporation 
"Issues in National Educational Data Collection" 

Jane L. David - Consultant 

"Improving the Quality and Utility of NCES Data" 

Eugene E. Eubanks - University of Missouri - Kansas City 
"Data Needs for Big City Schools" 

W. Vance Grant - NCES 

"An Elementary and Secondary School Statistics Program 
for the National Center for Education Statistics" 

Jane Hannaway - Princeton University 

"Two Suggestions for NCES Data Collection" 

Carole Hall Hardeman - ADROIT Publishing, Inc. 

"The Quest for Excellence/Pupil Self-Esteem 1 ' 

Forrest W. Harrison - NCES Retired 

"Review of Elementary/Seconda*y School Data: 

Needs of the National Center for Education Statistics" 

Willis D. Hawley - Peabody College, Vanderbilt University 
"Educational Statistics and School Improvement" 



Paul T. Hill - The Rand Corporation 

"The Politics of Educational Data Collection" 

Asa G. Hilliard XII - Georgia State University 

“Information for Excellence and Equity in Education" 

Maureen McClure and David N, Plank. - University of Pittsburgh 
"Educational Statistics for Educational Policy: 

A Political Economy Perspective" 

James M« McPartland, Henry Jay Decker, and Robert L. Crain - Johns Hopkins 
"A Model for N.C.E.S. Research on School Organization and 
Classroom Practices" 

Inabeth Miller - Gutman Library, Harvard University 
"A House of Bricks" 

Richard J. Murnane - Harvard University 

"Priorities for Federal Education Statistics" 

Gary Natriello - Teachers College, Columbia University 

"Products and Processes of the National Center for Education Statistice 
An Agenda for the Next Decade" 

Allan Odden - University of Southern California 
'•Federal Collection of School Finance Data: 

New Needs for an Era of Education Reform" 

Penelope L. Peterson - University of Wisconsin-Madison 
"The Elernentary/Secondary Redesign Project: 

Assessing the Condition of Education in the Next Decade" 

Valena White Pllsko and Alan Ginsburg, U.S. Department of Education (0P8E! 

Stephen Chaikind, Decision Resources, Inc, 

"Education Statistics: Assessing National Data" 

Elizabeth R. Reisner - Policy Studies Associates 
"New Areas for Educational Data Collection: 

What Students are Taught and What They Learn" 

Susan J. Rosenholtz - University of Illinois, Urbana-Champaign 
"Needed Resolves for Educational Research" 

Diane Scott-Jones - North Carolina State University 

"Assessing American Education: Shrinking Resources, Growing Demands" 

Ramsay w. Selden - National Institute of Education 

"Educational Indicators: What We Need to Know That We Don't Know Now" 

Marshall S. Smith - University of Wisconsin-Madison 

"Thoughts on Improving the Quality and Utility of NCES Data" 



E. Thomas - Johns Hopkins University 
"Issues and Considerations for a Ten-Year Program on 
Gleraentary and Secondary School Data Collection" 

nda J. Turnbull - Policy Studies Associates 

"Comments on an Elementary and Secondary Education Data Program' 1 

iael D. Usdan - Institute for Education Leadership 
"Educational Data Needs for the Balance of the 20th Century: 

Some Perspectives on the Emerging Environmental Context" 

ael Valdivieso - Hispanic Policy Development Project 
"Hispanics and Education Data" 

bert J. Walberg - University of Illinois at Chicago 
"National Statistics for Improving Educational Productivity" 

se Cherry Wilkinson - City University of New York 
"Assessing Students' Social and Communicative Achievement in School" 

WZATIONS 

art G* Lehnen - Indiana University 

iting on behalf of the American Statistical Association) 

"Educational Statistics for Studies of Policy and Administration" 

relation of American Publishers - a letter 

rican Association for Counseling and Development - a letter 

icil of Chief State School Officers - written comments 

- a letter 

'iorman Sims and Deborah A. Gona - The Council of State Governments 
"Assessing the Education Statistics Information Needs of 
'Jon-SEA Public Policy Decision Makers" 

Liam W. Turnbull - Educational Testing Service (writing on behalf of ETS) 
"Needs for Data in Education" 

aeran Church - Missouri Synod - a letter 

Lonal Center for Research in Vocational Education 

"Data on Vocational Education: Problems and Recommendations" 

onal Education Association 
"Suggestions for the NCES Redesign Project" 

onal Governors Association 
"Comments on the NCES Redesign 11 


Bruce Barker - Texas Tech University 

(writing on behalf of the Rural Education Association) 
"Research and Data Needs for Small/Rural Schools" 


FEDERAL AGENCIES 

Equal Employment Opportunity Commission - a letter and attachments 
National Science Foundation - a letter and attachments 


An additional paper cited by several authors, having been mailed to them as p 
of their background on NCES, is: 

Charles Cooke, Consultant, Apple Computers 
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>ign its elementary ana seconaary eaucdTion oaia program, rapers ana 
merits are being invited from education and other organizations, education 
rts and data providers, policy analysts, and users including representatives of 
ol districts and private schools, State executives and legislators, and the 
:utive and Congressional Branches of the Federal Government. The redesign 
?ct, leading to proposed revisions in the program of data about the nation's 
entary and secondary education activities, is using a process which is very 
i and very public, with all products to be published and circulated widely. 

The objective of the project is to design a 10-year plan of data collec- 
from institutions and individuals to be implemented beginning in the fall of 
>. This new plan will indicate the cross-sectional and longitudinal data 
ant to future and existant policy issues, and instructional and administrative 
s, as well as to measurement of our Nation’s education system. 

REDESIGN PROCESS will include: 

• a call lor invited papers and other comments from a variety of 
sources on data needed by educators, policymakers, and the public to 
address emerging and continuing issues in elementary and secondary 
education, and changes to current data collections to increase their 
usefulness; 

• a (mid summer) synthesis of these papers, extracting the essence 
of the papers to be widely circulated to stimulate public debate; 

• a period of public discussion and comment; 

• a concurrent review by NCES of the sources of non-comparability 
and inaccuracy in the current education data collections; 

• a later (early fall) expansion of the synthesis to include written 
public comment, accompanied by proposed strategies for acquiring 
data more responsive to the expressed needs; 

• regional hearings in November, to address how well concerns of data 
users and providers are addressed; 

• the publication of all the papers received from individuals and from 
organizations; and 

• the design of a 10-year plan of data collections from institutions 
and individuals about public and private schools, teachers, and 
students (with a preliminary draft circulated prior to the public 
hearings, and a final "draft"published in the Federal Register in 


of comments of interest to the redesign project. Further, they were asked for 
possible survey questionnaire items, specific measures, or indicators that could 
improve future analyses, and changes to existing procedures, definitions, and 
coverages. 

The guiding questions were: 

1. What data or data series are needed to support deliberations on future 
policy issues, or decisions on instructional and administrative needs, 
during the remainder of the 20th century ? 

(Link the issues or needs to the data items.) 

2. What additional data or data modifications ■■ in items, measures, in¬ 
dicators, or sample universes/frames •• would improve the utility, 
validity . or reliability of current national data files? 

(Identify the data files and how they would be improved.) 

3. What current NCES data series are most important to maintain and 
why? 

4. What current data elements or series are recommended to be deleted 
from current data programs and why? 

5. What other suggestions are offered for improving the relevance, tech¬ 
nical quality, and utility of the NCES data programs? 

ABOUT THE PAPERS 

The charge given to all authors and organizations preparing papers was 
very general in nature. All were encouraged to go beyond any one specific issue 
or area of major personal concern. Authors have, in fact, been encouraged to 
represent the breadth of issues in elementary and secondary education. 

For this review -• "Elementary and Secondary Education" can include edu¬ 
cational experiences from birth through the transition to postsecondary 
education or the workplace. The review can include all educational ex¬ 
periences: public or private, in “schools" or other locations, organized or 
not. 

For this review - "National Data" can include reference to any data col¬ 
lections, existing or proposed, and need not be confined to NCES data 
programs. For example, the paper can discuss any Federal Government or 
other national education data programs, such as the National Assessment 
of Educational Progress (NAEP), the Survey of Local Government Fi¬ 
nances (Bureau of the Census F-33 Series), or the Decennial Census of 

Pnniilatmn anrl Wrviicinn 



THE ELEMENTARY/SECONDARY EDUCATION DATA ACQUISITION PROGRAM 


National Center for Education Statistics 


The data sources and data sets that comprise the current NCES program are 
described in the material that follows. 

The three principal ways the Center acquires data are: 

1. Contractual agreements with data sources such as State education 
agencies (SEA's) under which the sources compile data from 
administrative records into specified reporting formats, 

2. Voluntary, self-administered sample surveys conducted by mail, and 

3. Interagency agreements with other Federal agencies under which these 
other agencies provide specified data sets and tabulations. 

Ihe various components of the current acquisition program are described below 
in terms of the population of inquiry, coverage, source, summary level, 
periodicity, and the data set included in each. 

I. Common Core of Data (CCD) — through 1984-85 


The Common Core of Data is the primary source of basic statistical data 
about public elementary and secondary educational institutions. Much of 
the data obtained are derived from administrative records maintained by 
the SEA 1 s. Each SEA compiles these data into the prescribed formats and 
transmits these reports to the Center per contractual agreements with 
each State. 

Part I, Public School Universe 


Population of inquiry: 

Coverage : 

Source : 

Summary level : 

Periodicity : 

Data set ; 


Public elementary and secondary schools in 
operation that school year 
Census 

State Education Agency administrative records 
Schools aggregated to Local Education Agencies 
and States 

Annual update: new schools added and closed 
schools deleted 

Identity* of Local Education Agency that operates 
the school 

School name and address 
Fall membership 

Full-time equivalent number of classroom teachers 
Type of operation 
Type of school 
Grade span 




Part II, Local Education Agency (LEA) Universe 


Population of inquiry: 


Cove rage 
Source 

Summary level 
Periodicity 
Data set 


Local education agencies (as defined in the 
Education Consolidation and Improvement 
Act, PL 97-35) 

Census 

State education agency administrative records 
LEA's aggregated to States 
Annual update 

Identification number that links LEA to other CCD 
responding units 
Name and address of agency 

operating status: does or does not operate a 
school 

Fiscal status: independent of or dependent upon 
a parent government for spending authority 
Control status: Board elected or appointed 
County, SMSA, and metropolitan status codes 
Boundary change indicator (newly formed or 
reorganized) by year of change 
Agency type code (local school district, 

supervisory union, regional education service 
agency, etc.) 


Part III, Local Education Agency Nonfiscal Report 


Population of inquiry: 


Coverage 
Sour ce 

Summary level 
Periodic! ty 
Data set 


Local education agencies as (defined in the 
Education Consolidation and Improvement 
Act, PL 97-35) 

Ce nsus 

State education agency administrative records 

LEA's aggregated to States 

Annual 

Agency identification number 

Full-time-equivalent number of (Pre-K, K, 1-12) 
teachers, instructional aides, and other staff 
Membership (pre-K, K, 1-12) 

Number of schools operated by the agency 


Part IV, Public School District Finance Report 


Population of inquiry: Local public school districts (Regional education 

service centers and other LEA's are excluded 
from coverage.) 


Coverage 

Source 

Summary level 
Pe riodici ty 


Census 

State education agency administrative records 
Local school districts aggregated to States 
Annual 





: IV. continued: 


i set 


V, State 


lation of 

rage 

ce 

ary level 
odici ty 
set 


VI, State 


lation of 

rage 

ce 

ary level 
dicity 


: Revenues by source (local, intermediate, State, 

or federal) 

Current expenditures by major function 
(instruction, support services, and 
noninstructional services) 

Average Daily Attendance (ADA) 

Other uses of funds (debt service, construction, 

etc. ) 

Special exhibits including amounts received from 
property taxes, tuitions and intergovernmental 
transfers, amounts spent for salaries, 
interest on debt, employee benefits 

Aggregate Nonfiscal Report 

inquiry: State education agencies 

: Census 

: State education agency administrative records 

: State 

: Annual 

: Full-time-equivalent (FTE) number of employees 

by major assignment category: 

Instructional Staff: Pre-K, K, 
elementary, secondary, ungraded 
Instructional Aides 
Counselors 
Librarians 
Adtni n is tra to rs 
other support staff 

Fall membership by grade-level groupings: 

Pre-K, K, 1, 2.. 12, ungraded 

Number of high school graduates from day 
programs and from other programs during 
preceeding year 

Aggregate Fiscal Report 

inquiry: State education agencies and other State agencies 
that provide resources to support LEA'S 
: Census 

: State education agency administrative records 
: State 
: Annual 





Part VI. continued: 


Data set ; State aggregate of: 

Local School District revenues by source 
(local, intermediate, State, and federal) 
School district current expenditures by major 
function (instruction, support services, and 
non-instructional services) 

Other agency current expenditures for and on 
behalf of school districts by major function 
Special exhibits including expenditures for 
employee benefits and other fixed charges 
Average daily attendance (regular term and 
summer FTE) 

Expenditures for non-instructional services 
(food services, enterprise activities) 

11, Sample Surveys 


In addition to the types of administrative data collected through 
Common Core of Data, the Center conducts a series of sample surveys to 
obtain other data on public and private elementary and secondary 
education. These surveys are described below. 

Private School Survey 


The content of the periodic private school surveys always includes a 
core of school summary data. Supplements to this core are designed to 
obtain detailed data of current interest to address emerging policy 
issues. Private school surveys are planned for school years ending in 
even numbers. 

Population of inquiry: Private elementary and secondary schools, 

excluding preprimary (only) schools. 

Coverage (1983-84) : Nationally representative list sample of 1500 

schools, supplemented with a sample from 
additional schools found by canvassing a 
sample of 75 geographic areas. 

Private school administrators (future surveys 
may include response from teachers, pupils, 
or parents) 


Source 



Private School Survey continued: 


Summary level : School aggregated to national estimates 

Periodicity : Biennial (school years ending in even numbers) 

Data set (1983-84 ) : Fall membership by specified grade categories 

Full-time-equivalent number of employees by 
major assignment category 
Number of teachers by: 
highest earned degree 
years of experience 
selected salary intervals 
Tuition rates charged by instructional level 
Specified program offerings and student 
enrollments in each 

Estimated revenue from specified federal 
program sources and student participation 
Selected school characteristics such as 
admission requirements, disciplinary 
policies, length of day and school year, etc 
Number of high school graduates in preceding 
year 

Public School Survey 


The content of the periodic public school surveys will always include 
core summary data. Supplements to this core will be designed to 
obtain more detailed data of current interest to address emerging 
policy issues. These surveys are planned for school years ending in 
odd numbers. 


Population of inquiry: 
Coverage (1984-85) : 


Source 

Summary level 

Periodicity 
Data set (1984-85) 


Public elementary and secondary schools 
Nationally representative sample of 2800 
schools, and approximately 11,000 teachers 
selected from the sample schools. 

School administrators and teachers 
Schools and teachers aggregated to national 
estimates 

Biennial (school years ending in odd numbers) 
School administrator questionnaire items: 

Fall membership 
Design capacity of the school 
Minority enrollment as a percent of total 
enrollment 

Full-time-equivalent number of teachers and 
other employees by major assignment 
category 

Grade span of pupils served 
Class size by major subject area 




Public School Survey continued: 


Data set (1984-85) : Number of high school graduates 

School average SAT/ACT scores and percenl 
seniors tested 

Number of volunteers by activity category 
Information on teacher incentive plans 
Information on computer use 
Teacher questionnaire items: 

Highest earned degree 

College credits by subject matter field 

Information on additional training 

Years of experience 

Teaching assignments 

Amount of homework assigned 

Use of teacher aides and volunteers 

Summer employment 

Detailed information on hours spent durii 
week on specified activities 
Compensation and incentives 
Age, sex and racial-ethnic affiliation 

Recent College Graduates Survey 

This survey obtains data on employment and earnings of persons who 
received baccalaurate or masters degrees in the preceding year. A 
component in this survey obtains more detailed data about graduate: 
sought and/or found employment in schools or school districts. 

ry: Recent college graduates receiving bachelo 
and masters degrees 

: Nationally* representative sample of 400 
colleges and universities, and 10,000 
bachelors and 2,500 masters degree 
recipients 

: Individual graduates 

: Individuals aggregated to national estimate 

: Occasional 

: Core data includes: 

Age 

Year of degree 
Type of degree 
Awards 

Current employment 
Occupation 
Salary 

Specific assignments (if teaching) 


Population of inqu 
Coverage (1984-85) 


Source 

Sumioary level 
Periodicity 
Data Set 



Survey of Teacher Demand and Shortage 


This survey obtains data on the number of teachers by assignment, the 
number of new hires, the number of positions that could not be filled, 
and recruiting and employment practices. 


Population of inquiry: 

Coverage (1983-84) : 

Source : 

Summary level : 

Periodicity : 

Data set (1983-84) 


Local school districts, State operated schools, 
and private schools 

Nationally representative sample of 2,540 
LEA's and 1,000 private schools. 

School administrators 

School district or school aggregated to sub¬ 
national and national estimates 

Occasional 

Number of budgeted teaching positions 

Number of shortages by subject matter 

Number of continuing teachers filling positions 
by certification status 

Number of new hires 

Matrix format description of teacher incentive 
plans 

Number of PTE teacher positions by subject 
matter assignments, and by certification 
status of incumbents or shortage status 

Region, size, and metropolitan status (for 
public school districts) 


High School and Beyond 


High School and Beyond is a national longitudinal study of cohorts of 
1980 high school sophomores and seniors. Questionnaires and cognitive 
tests were administered to students. Follow-ups are planned to 
determine what happened to these students after high school completion. 


Population of inquiry 
Coverage 

1980 Base Year 


1982 Follow-up 


High school students, their parents, teachers, 
and school administrators 
Nationally representative sample of 1,015 high 
schoois 

30,000 1980 seniors 
3 0,000 1980 sophmores 
10,370 teachers 
1,015 administrators 
3,700 parents per cohort 
11,227 1980 seniors (a subsample of the 1980 
senior cohort) 

27,118 1980 sophmores 



High School and Beyond continued. 


Source 

Summary Level 

Periodicity 
Data set 


School administrators, students, teachers, and 
parents 

Individual respondent aggregated to national 
es tima tes 

Occasional re-contact of each cohort 
Self-reported student characteristics 
Self-reported student opinions and aspirations 
Cognitive test scores 
Self-reported student activities 
High school transcripts for 1980 sophomores 
Postsecondary transcripts of 1980 seniors 
Scholastic Aptitude Test scores for 1980 
seniors 

Armed Services Vocational Aptitude Battery 
scores for 1980 seniors 
Student financial aid and guaranteed student 
loan records for the 1980 cohort in 
postseconday institutions for each of four 
years following high school graduation. 


Library/Media Center Survey 


This survey obtains data about library and media centers serving 
elementary and secondary schools, as well as other library facilities 
and services. 


Population of inquiry: 

(1985-86 survey) 
Coverage : 


Source 

Summary level 

Periodicity 
Data set 


School library/media centers 

Nationally representative sample 
1,7 00 private schools 
4,500 public schools 
Library/media center administrators 
Library/media center responses aggregated to 
national estimates 
Occasional 

Number of holdings by major category, both 
titles and volumes 

Descriptive information on services provided 
Full-time-equivalent number of employees 
by major assignment category 
Expenditures for acquisitions 



III. Other Federal Agency Data 


The Center enters into interagency agreements to obtain data that can 
be more efficiently acquired by "piggy-backing" surveys conducted by 
other Federal agencies, such as the Center's acquisition of data from 
the Current Population Survey conducted by the Bureau of the Census. 

The utilization of such surveys permits the Center to obtain data that 
would otherwise require independent, complex, multiple-stage, and 
costly survey designs. 

Acquisitions that employ this mechanism vary from year to year 
depending upon need and budget contraints. The single survey described 
below is one that has been employed for a number of years to establish 
a data time series. Other data sets have been obtained in other 
similar efforts, principally on a one-time basis. 

Preprimary Enrollments of Children 3-5 years old 

The October Current Population Survey obtains data about the households 
surveyed, and educational and demographic characteristics of all 
household members. The survey also collects, as is its primary monthly 
purpose year round, labor force data for household members 14 years of 
age and older, A supplement to the October survey obtains data on all 
household members enrolled in schools or colleges. 


Population of inquiry: 

Cove rage : 

Source : 

Summary level : 


Data set 


Households and individual members 

Nationally representative sample of 65,000 
house holds 

Household respondents 

Households and individuals aggregated to 
national, regional, and other subnational 
estimates 

Family or household characteristics including 
composition, income, and other socioeconomic 
descriptors 

Occupational and labor force status of persons 
14 years and over 

Age, sex, and race/ethnic origin of all 
individuals 

School enrollment of all children and adults, 
if any enrolled, by grade level, control 
of institution, full or part-time attendance 
status, and type of program (including 3-5 
year olds, the focus of some NCES reports) 



